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1 BACKGROUND 


Air transportation, the most rapidly growing mode of transportation, is also one of the safest modes 
of travel. Nevertheless, the public demands that safety levels continuously improve and that the 
absolute number of aviation accidents continues to decline, even as air traffic levels increase. 

NASA’s Aviation Safety Program (AvSP) was initiated in 2000 to develop the enabling 
technologies that could, if implemented, reduce the aircraft accident rate by a factor of five within 
ten years and by a factor of ten within twenty years. One of the projects within the AvSP, the 
Aviation System Monitoring and Modeling (ASMM) project, addresses the need to provide decision 
makers with the tools for safety improvement by identifying and correcting the predisposing 
conditions that could lead to accidents. The objective of the ASMM project is to develop 
technologies that will enable proactive management of safety risk from a system- wide perspective. 1 

A proactive approach to identifying and alleviating life-threatening conditions in the aviation system 
entails a well-defined process of identifying threats, evaluating causes, assessing risks, and 
implementing appropriate solutions. This process is not a trivial undertaking. It requires continuous 
monitoring of system performance in a non-punitive environment; learning from normal operational 
experience; comparing actual performance to expected performance; identifying the precursor events 
and conditions that foreshadow most accidents; designing appropriate interventions to minimize the 
risk of their occurrence; and having a system in place to monitor the efficacy of the interventions. 

At each of these stages, airline domain experts, air traffic managers, and other providers of aviation 
services must make key decisions. The ASMM project provides computational tools that focus the 
attention of human experts on the most significant events, and that help them identify the factors that 
distinguish unsafe operations from routine operations. The purpose of the ASMM tools is to convert 
a bounty of raw aviation operational data drawn from many sources — aircraft flight data recorders, 
ATC radar tracks, maintenance logs, weather records, and aviation safety incident reports — into 
insightful interpretations of the health and safety of the National Aviation System (NAS). 
Computational sciences and information technology can be used to cope with the great disparity of 
formats and types of these data sources that include continuous and discrete quantitative data and 
textual data. However, when it comes to making decisions about aviation safety, the best use of 
computer sciences is to help the human gain insight into operations. Even the most advanced 
adaptations of information technologies and computer sciences cannot replace the human expert 
with automated decisions. 

Identifying and recognizing precursors of the next accidents pose considerable challenges that are 
being addressed in many domains, including nuclear, medical, chemical, pharmaceutical, and space 
missions, as well as aviation (Phimister 2003). Each domain may have its own definition of a 
precursor, and so it is important for the reader to understand what the authors mean when we use the 
term throughout this report. 


1 We use the term “system” in this report to include all aspects of air transportation including, but not limited to, air 
carrier and general aviation operations, air traffic management, training, maintenance, design, manufacturing, rules 
and regulations, weather, and organizational cultures. 
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We use the term precursor to mean the symptom of a systemic problem that is conducive to human 
error and that, if left unresolved, has the potential to result in an accident. A symptom is a 
measurable deviation from expectations or the norm. It is the problem that must be treated, not the 
symptom. 

We need to start with a model to understand the problem. In Section 5, we describe our concept of 
Scenario as follows: 

SCENARIO = {CONTEXT + BEHAVIOR OUTCOME} 

We consider the Context to be that of the last safe state 2 , and the Behavior results in the transition to 
the Outcome. When the Outcome is an anomalous (unwanted or compromised) state, the last safe 
state is identified as a precursor. A set of our experiments (the first of which is described in 
Section 8) is directed at exploring the correlations between the categorical features of the Context 
and of the anomalous Outcomes. Our assumption (yet to be proven) is that such correlations, 
together with inputs from domain experts, will help us identify those specific categorical features 
that cause the last safe state to be a precursor. That is, the causal factors of the Behavior produce the 
transition from the last safe state to the unwanted Outcome. Our concept of Scenario constitutes the 
basis for the experiments discussed in Sections 7 and 8, and the future directions of this research 
discussed in Section 9. 

The tools developed to date under the ASMM Project for extracting information from data — in 
particular, quantitative data — have been based largely on statistical analyses. However, as discussed 
by Pearl (1997), statistical correlation is a necessary but not sufficient condition for causality. 
Correlations among precursors, anomalous states, and incidents can provide, at best, partial and 
indirect evidence about causal links. For example, when we explore correlation between anomalies 
and contextual factors as described in Section 8, we must be careful that we do not lead the reader to 
believe that these are necessarily causal factors. An example is that incidents associated with 
Traffic-alert and Collision- Avoidance System (TCAS) alerts will be well correlated with an aircraft 
being airborne, but we should not conclude that being airborne is a causal factor of TCAS events. A 
further caution is that we must not come to conclusions based on statistical analyses that may not 
have included significant operational factors (Simpson’s paradox). For example, a comparison of 
operations at two airports may be skewed quite differently if we include weather. Or to cite another 
example, a comparison of operations of two make/model aircraft (say, the Boeing 737 and the 
Airbus 320) may be changed if we included aspects of the flight crews’ training experience. Causes 
cannot be derived on the basis of statistical or functional relationships alone. 

It is important that the precursor not be viewed as being synonymous with causality. Recall that we 
have identified the last safe state as a precursor if (and only if), among its categorical features, it 
includes those particular factors that cause the Behavior to produce a transition to an anomalous 
state. It is in this sense, that we use the term “causal factors” to include: 

• Conditions necessary for the occurrence of a precursor 

AND 


9 

~ When we speak of the “state” in this report, and, in particular, in Section 5, we mean the state of the entire system. 
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• Conditions that increase the probability of occurrence of that precursor. 

Note that the treatment of the causal factors often entails a re-design, a new procedure, and new 
training. 

Consequently, in the approach to the study reported here, we are searching for the causal factors of 
the precursor incident and not for the anomalous consequences, per se. Even more precisely, as 
discussed in Section 9, we are seeking to uncover those particular causal factors of the precursor that 
explain why the transition to the anomalous state occurred. 

Identifying the precursors of the next accident is a particular challenge in a complex operating 
environment like that of aviation with many interacting components. People are key components of 
the aviation system, and human error is frequently cited as a major contributing factor or cause of 
incidents and accidents. Sixty to eighty percent of fatal aviation accidents are attributed (rightly or 
wrongly) to human error. (See, for example, Boeing 2002 and 2004.) However, simply saying that 
one or more of the humans in a system may have made a mistake is not constructive. 

The attribution of “human error” is a social- and psychological-based judgment of human 
performance made in hindsight that is invariably biased by knowledge of the outcome (Woods et al. 
1994). However, human performance is relied upon to resolve uncertainties, conflicts, and 
competing demands inherent in large, complex systems. Consequently, human performance is as 
complex as the domain in which it is exercised and cannot be judged independently. Human 
behavior is context-dependent, and little can be understood of the causes of human error without 
understanding the prevailing as well as the more distal precursor conditions conducive to error. 
Much depends on being able to determine how complex systems have failed and on the human 
contribution to such outcome failures. Consequently, our question is, “Why do professional, well- 
trained, highly motivated operators of the aviation system make mistakes?” Our focus is on 
uncovering and understanding those precursor conditions that elevate the probability of downstream 
human errors and that, in turn, may contribute to aviation safety incidents or accidents. Knowledge 
about these systemic features helps us to understand how they shape human behavior and how to 
improve the performance of the system. 

The ASMM project has developed automated tools for extracting information from both quantitative 
numeric data, and from qualitative textual data, and for recognizing information from either data 
source that may be relevant to a particular query. Information extracted from quantitative data 
sources helps the domain expert understand the objective aspects of what happened, and from 
qualitative data sources to understand the subjective aspects of why an incident occurred. Each 
ASMM tool contributes insights into the complete picture of an event, and supports the 
complementary processes of causal analysis and safety-risk assessment. Causal analysis and safety- 
risk assessment, together with analysis of associated costs and benefits, are all required in order for 
experts to formulate appropriate interventions. 

This report describes a conceptual model and an approach to automated analyses of textual data 
sources that primarily aid the expert in understanding why an incident occurred. Throughout this 
report, we will make reference to extracting objective as well as subjective information from textual 
reports. However, our main focus is on understanding why an incident occurred, for which we must 
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rely on the subjective perspective of the reporter of the incident. We rely on other quantitative data 
sources (e.g., in-flight-recorded data and air-traffic-radar data) and other ASMM tools (Chidester 
2001, Chidester 2003, Ferryman 2001, and Statler et al. 2003) to extract the complementary 
information about what happened. 


2 THE CONTEXT OF THIS STUDY 


This report explores a first-generation process for routinely searching large databases of accident or 
incident reports, and consistently and reliably analyzing them for causal factors of human behavior 
in aviation operations (the why of an incident). Incident reports indicate the presence of problems in 
systems that, if left unresolved, have the potential to result in an accident (Heinrich 1959). The 
experiential account of the incident reporter is the best available source of information about why an 
incident happened. 

The analysis of textual databases poses several challenges. First, the process is typically labor- 
intensive and requires high-priced domain expertise. Further, such analyses not only require experts 
from aviation operations to understand what happened according to the reported incident, but they 
will often also require experts in human factors to explain why events happened. Unfortunately, 
current methods for analyzing textual data often focus on what went wrong and what the 
consequences were, but fail to exploit this primary source of information about why an event 
happened. Therefore, there is a need for new analytical methods and automated capabilities to help 
the experts mine these rich and complex textual databases for insight into the causal, contributing, 
and aggravating factors of an event. 

There are two primary sources of aviation experiential, textual reports to which reference will be 
made throughout this report. One of these is the database of the Aviation Safety Reporting System 
(ASRS), (Reynard et al. 1986, Chappell 1997, and Connell 1999), which is a collection of nearly 
1 15,000 narratives of aviation safety incidents that have been voluntarily submitted by reporters 
from across the aviation industry. The ASRS, managed by NASA and funded by the FAA Office of 
System Safety since 1976, is one of the world's best-known and most highly regarded repositories of 
safety information. ASRS incident reports have been used extensively for this study, as well as for 
earlier studies, as a unique “test bed” for evaluating the tools that are being developed under the 
ASMM Project for processing and analyzing textual data. In addition, this study has benefited from 
convenient access to knowledgeable ASRS personnel who have developed, operated, and utilized 
the system to the benefit of the industry. 

The Aviation Safety Action Programs (ASAP) are currently generating the other database of textual 
incident reports on which ASMM tools and methodologies are being tested. The ASAP programs are 
intramural, voluntary safety reporting programs through which certificated personnel (pilots, 
dispatchers, mechanics) at participating air carriers report any safety concern they observe, even if it 
resulted from their own errors. Modeled on concepts and principles first put into place under the 
ASRS, ASAP reporting and processing are non-punitive and confidential. Under a collaborative 
agreement with one air carrier, some of the tools and methodologies described in this report are 
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being tested on ASAP reports, but this report will address only the experiments that used ASRS 
incident reports. 


This report describes a data model and related experiments aimed at achieving an automated 
understanding of the causal factors of the human error embodied in reported incidents. Our approach 
is not designed to fit any specific incident-reporting system — our methodology must be sufficiently 
generic to be used with any database of textual reports of aviation incidents — but we are going to 
refer often to the ASRS because it is representative of all such databases. 

There are, of course, many tools already available for searching textual databases. We have 
considered their applicability to our needs and, in Appendix A, we describe several tools that we 
tested on subsets of ASRS reports prior to undertaking this study. Others are continually being 
identified and evaluated in this ongoing study. 

2.1 Research Objectives 

The intent of this research is to better understand the quantitative and qualitative attributes of an 
aviation incident, and to identify the respective contributions of their interaction to incident 
occurrence. The focus of our research is on the contextual aspects of this interaction. Our specific 
research questions are as follows: 

1 . What is the fundamental structure underlying an aviation incident? What are the contextual 
parameters associated with each part of this structure? Which of these parameters can be 
considered objective (based on observable data), and which subjective (existing largely within 
the reporter’s mind)? Can the parameters used to define the structure of an incident be adapted to 
an automated clustering process? 

2. What are the pragmatic constraints we must consider in undertaking experiments in automated 
clustering that are based on statistical analysis and that can be used on very large databases of 
textual reports? Can the similarities highlighted by an automated clustering process be checked 
for validity and usefulness? How can we use domain knowledge to minimize the domain size 
that the automated tools must consider? 

3. Is there a conceptual paradigm that will allow us to explain the sequential, discriminating factors 
that constitute the why and how of incidents in large aviation databases like the ASRS database? 
Can this description be used to “tune” automated analyses that will identify contextual 
similarities between groups of incidents? 

2.2 Outline of This Research Report 

Incident reports are written by operational personnel who try to describe as clearly as they can a 
situation they encountered having safety implications. The report forms used by the ASRS and the 
ASAP contain both fixed fields and free narratives. Together they present the story of what 
happened, how it happened, and, often, some attempt to explain why. 

Automated tools for aiding analyses of textual databases must enable efficient retrospective search 
for any prescribed event, and must also enable discovery of the un-envisioned. One of the biggest 
challenges of proactive management of risks is to develop the capability to explore data without 
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knowing what we are looking for. The automated tool should be able to extract typical incidents, but 
it should also be able to highlight atypical ones and describe their differences in a way that is similar 
to the tools we have developed for analyzing flight-recorded quantitative data (Ferryman 2001). 
Fundamentally, both of these requirements rely on a capability to extract groups of similar incident 
reports from a very large database. 

In the development of tools for analyzing flight-recorded data, we found that a concept we called the 
“flight signature” was useful in guiding the automated identifications of similarities among flights. 
We believed that we needed a similar model to capture the underlying structure of an incident report 
and to guide the automated analyses of textual data. Our many years of experience reading almost 
600,000 ASRS reports 3 led us to developing a model based on a sequence of states and transitions. 

In Section 3, we will describe the resulting incident model to aid in understanding the structure of 
incidents. 

As stated previously, the incident-report forms have fixed fields that the reporter is expected to 
complete as well as the narrative in a free-text field. The fixed fields (i.e., the attributes), together 
with the information the reporter and the ASRS analyst enter into those fields (i.e., the values of the 
attributes), comprise the codification of the incident report. A coordinated exploitation of the 
information extracted from both the fixed fields and the free-text fields is necessary to achieve our 
objectives, and this is discussed in Sections 4 and 5. 

Section 5 introduces the concept of a scenario, a simplified subset of the incident model, as a 
pragmatic approach for guiding automated clustering of similar incidents. 

Section 6 presents a summary of the results of the studies described in the previous sections to lay 
the groundwork for the discussions and experiments in the following sections. 

Section 7 describes a limited experiment based on 40 incident reports. The aim was to evaluate what 
could be expected from a clustering process based only on the taxonomy (i.e., attributes and values) 
of the fixed fields and the concept of scenario. 

Section 8 describes a preliminary experiment to evaluate the capability to identify and cluster 
reliably on similarities of what happened. 

Section 9 describes the approach to extract information about why the event occurred, after 
automatically identifying what occurred, using the scenario model. Our approach is based on the 
proposal that loss of situation awareness is the behavioral failure primarily responsible for errors in 
our aviation world. 

Section 10 presents a summary of the conclusions of this study and projections of the continuing 
studies. 


Although the ASRS has received about 600,000 reports in its 28 years of operation, as indicated previously, only about 
1 15,000 of these have been entered into the database. This has been the result of many factors including judgments on 
significance and multiple reports on the same incident, but, mostly, limitations of available funding. 
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3 THE INCIDENT MODEL 


The management and the exploitation of very large databases of incident reports highlight the need 
for sophisticated tools to process free text and to automatically classify reports in a way that is 
meaningful to experts. In order to extract useful information (whether quantitative or textual) from 
large databases, it is necessary to identify global patterns and relationships to aid decision-making. 
We need a model with which to guide the automated analyses. 

Report narratives embody naturally occurring chains of events and the transitions between events. 

An incident report is similar to a script of a play: it describes the environment of the action, the 
protagonists, and the steps (or acts) in the course of the story. In fact, we will introduce the concept 
of a “scenario” of a report to represent these naturally occurring “stories.” The aim of the next 
section is to define more precisely the elements of this model, in order to highlight concepts that 
could be used to calibrate automated tools that will extract information about why the incident 
occurred. We will first define the meanings of the words used in our model. 

3,1 Definitions 

World: “ An area, sphere or realm ( afield of interest or study ) considered as a complete 
environment” (Collins Dictionary) 

The world of an ASRS-like incident report is composed of all the aircraft, people, weather elements, 
ground equipment, and other factors that have to be taken into account to describe and understand 
the course of events that happen (the story). The world of the aviation incident report is a subset of 
the real world . The boundary of this world (i.e., defining what is needed for a complete environment) 
is always subjective and depends on the granularity of the description and the scope of the 
understanding that we want to reach. Following are two examples of what we mean by a bounded 
world of an incident report: 

1. A report describes an incident that took place in San Francisco in severe weather conditions. The 
cause of such weather conditions is the El Nino phenomenon. Nevertheless, El Nino will not be 
part of the world of this incident, as the aim is not to understand the weather conditions, but the 
incident that occurred. 

2. A report is about an incident in which an air- traffic controller made an inappropriate decision 
that resulted in a loss of desired separation between two aircraft. Which aircraft have to be taken 
into account in the world of this incident? Only the ones in conflict? All the aircraft under the 
control of the controller (because they change the task of the controller)? We will consider only 
the aircraft in conflict, as all of the aircraft under the controller’s direction are outside the bounds 
of our “story.” 

The world is changing continuously; therefore, its description has to be dynamic. 

Parameters: The world and its evolution are observed and described through parameters. Some 
parameters are linked to physical measures; others are not. Parameters can be objective (i.e., defined 
on bases of observable data) as, for instance, flight altitude, or subjective (i.e., the value of the 
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parameter is not a result of observable data) as, for instance, the ASRS report-form entry “passenger 
misconduct” to describe a world in which a passenger exhibits abnormal behavior. 

The choice of parameters adopted to describe the world is crucial, as this choice largely determines 
which sort of automatic treatment will be possible. Two criteria have to be considered in the 
selection of parameters: the level of abstraction of the concept embodied by the parameter, and the 
ability of this parameter to represent “small” variations. 

For instance, let us consider a very simple world that consists only of an electric light bulb. We can 
choose to describe this world using the terminal voltage of this bulb, or with a Boolean parameter 
(i.e., on/off). The levels of abstraction of these two factors are not the same, and their abilities to 
describe the evolution of the world for short periods of time are not the same. If we want to have 
only a “high level” understanding of the evolution of the world (as a pilot might), the Boolean 
parameter is the better choice. If we want to understand the response of the component to specific 
electrical signals, the voltage is a better- adapted parameter. 


ON 

OFF 



— 

— 

Volts 

— 

^ 



r 



TO T1 


Boolean Parameter 


Voltage Level 
Time 


Figure 1. Example of parameter. 


State (of the world): The state of the world is the description of the world at a fixed time (a 
snapshot). The state of the world is given by the values of all the parameters that have been chosen 
to describe that world. For instance, the state of the “bulb” world at time TO is “Off’ if we describe 
this world with the Boolean parameter. 

Event: “Anything that takes place, especially something important” (Collins Dictionary) 

When we observe the evolution of the world over a period of time, the evolution of some of its 
parameters can be described by an abstract concept called an event . An event is defined by a 
progressive evolution of a set of parameters that “makes sense.” For instance, the evolution of the 
Boolean parameter between TO and T1 is typical and allows us to define the event: “switch on.” 

As an event describes the evolution of some subset of all of the parameters that describe the world, it 
gives information about the state of the world before the event, during the event, and after the event. 

The level of abstraction of the events defined can be a simple evolution of a few parameters (for 
instance, our event “switch on”), or a very complex combination of parameters (for instance, an 
aircraft “takeoff’). Some events, such as “takeoff,” can be decomposed into more elementary events 
(such as “accelerate,” “rotate,” “liftoff,” etc.), called sub-events. 
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Transition: “Change or passage from one state or stage to another” (Collins Dictionary) 


A transition is a combination of events that allows the world to change from one state to another. 

Incident: Aeronautical operations are planned according to some rules. These rules try to keep the 
world in “safe” states and provide criteria to define “unsafe” states (for instance, two commercial 
aircraft in flight separated by only 100 feet define an “unsafe” state of the world). When the 
evolution of the world brings it to some “unsafe” state, we have an incident. 

An incident is an evolution of the world such that the state of the world reaches some “unsafe” state, 
and then returns to a safe state. 


3.2 The State/Transition Representation of the Incident Model 

Reporters of aeronautical incidents describe problems encountered during flight operations. They 
usually tell them as stories and concentrate on what happened, on the involvement and behavior of 
people as well as on the important features that help us to understand why these problems occurred. 
Hence, we assume that the course of an incident is well described by a sequence of states and 
transitions, and that the whole incident can be decomposed into a sequence of transitions leading the 
world to evolve from one state to another. Our first assumption is that the essence of the evolution of 
our world can be captured adequately from the report of the incident, and that this model can be used 
to tune clustering tools. 

Therefore, all the information contained in the report can be associated with a description of a state 
of the world, or with the characterization of an event that contributes to a transition . 

Example: (ACN 81075) “WE WERE ON A VISUAL APCH BEHIND A WDB FOR 
RWY 28R. AT ABOUT 1000' AGL THE TWR OFFERED US 28L. WE CHANGED 
TO 28L AND THE TWR CLRD THE WDB TO CROSS 28L AHEAD OF US. THE 
WDB DELAYED XING AND WHEN WE WERE CLOSE IN THE TWR OFFERED 
US 28R. WE ATTEMPTED TO CHANGE TO 28R BUT WERE TOO CLOSE IN TO 
MANEUVER AND SO WE WENT AROUND.” 4 

In this example, the first sentence describes a state of the world, while the phrase, “We changed to 
28L,” describes an event that belongs to the first transition. The evolution of the world as described 
in this narrative can be represented by a sequence of 4 states, as shown in figure 2. 


4 For the reader who may not be familiar with aviation abbreviations, following is a literal translation: “We were on a 
visual approach behind a wide-body for runway 28 Right. At about 1000 feet above ground level, the tower offered us 
28 Left. We changed to 28 Left and the tower cleared the wide-body to cross 28 Left ahead of us. The wide-body 
delayed crossing and when we were close in the tower offered us 28 Right. We attempted to change to 28 Right but 
were too close in to maneuver and so we went around.” 
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Time 



Figure 2. A state/transition decomposition of an incident report. 


Nearly all stories (whether in the aviation world or not) can be decomposed into a State/Transition 
sequence. However, aviation incident reports have other characteristics that we need to exploit. The 
notion of an aviation incident implies an issue of safety: operational personnel consider that an 
incident occurs if, for a period of time, the situation is considered as “unsafe” or anomalous. Such 
notions are not clearly defined and not always interpreted in the same way by all operational 
personnel. Nevertheless, we can assume that, at a given point in time, aircraft are expected to exhibit 
a set of Required Properties that define the aircraft (viewed as a total system) as being in a nominally 
safe state. The Required Properties relate to the aircraft’s 

• position (altitude, latitude, longitude, airspace occupancy) 

• trajectory (heading and projected course) 

• flight dynamics (attitude, rotation rates, and speed) 

• airframe integrity 

• propulsion status 

• compliance with clearances, regulations, and SOPs. 

This definition implies that there is an expected reference value for each of these properties at every 
point in time. On the basis of these Required Properties, we can define the following three states of 
our aviation world: 

SAFE: All the aircraft and people (crews, traffic controllers...) and all the key systems (aircraft 
systems, ILS...) are in a state that approximates normalcy. 

COMPROMISED: A person involved in the situation or an aircraft system is in an undesirable 
state, or undesirable environmental factors impinge upon the aircraft, but for all the involved 
aircraft their Required Properties are still nominal and the separation between aircraft complies 
with norms. 

ANOMALOUS: One or more of the Required Properties of an aircraft or an involved person is 
observably not in compliance with pertinent norms. 
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We can now characterize what we call an aviation incident as follows: 


INCIDENT: An incident is a finite sequence of states and transitions such that: 

• the first state is safe, 

• the last state is safe, 

• all the other states are either compromised or anomalous, and 

• at least one state is anomalous. 

If the state does not return to a safe one, the story is not related to an incident, but to an accident. If 
no anomalous state is reached, the story is not considered to be an incident. (Figure 3 shows this 
description of our incident model.) As shown in figure 3, the last unsafe state is often an anomalous 
one, but this constraint is not required by our definition. 


SAFE state COMPROMISED or ANOMALOUS states SAFE state 



State 1 State 2 State n-1 State n Time 


Figure 3. The incident model. 


The aim of our incident model is to identify the key components that can be used to tune automated 
data-mining tools. We are proposing that this representation (i.e., the Incident Model), with its 
complete descriptions of all of the states and transitions during the evolution of an incident, is a 
generic model of any and all experiential reports of incidents that occur in our world of aviation. The 
descriptions of the states and transitions are based on parameters. Therefore, we will need to study 
the set of possible parameters, and we will discuss this in the next section. 


4 PARAMETERS FOR THE DESCRIPTION OF STATES AND 

TRANSITIONS 


We said in Section 3.1 that the world and its evolution are observed and described through 
parameters. In the example presented in figure 2, states and events have been described in an 
informal way by sentences or words extracted from the narrative. Choosing a set of parameters that 
describe a state of the world and a set of parameters that describe a transition will give us a more 
formal description of each part of such an incident. 
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A wide variety of taxonomic structures 5 is used in the different accident/incident databases. 6 Our 
first objective was to identify the taxonomy, together with all of its possible terms and their 
structure, which is most suitable for reports of incidents in our world of aviation. Each term must 
map to a parameter in the description of our world in the incident model. The set of parameters that 
we selected for our study is based on three taxonomic structures underlying three codification forms 
designed specifically for use with the ASRS database. 

• The ASRS codification: The codification of an ASRS incident report is comprised of the set 
of attributes (the fixed fields) and the values of those attributes (entered by the reporter). This 
structured set of ‘descriptors’ is currently used to describe the incident and store it in the 
database. The codification form is designed for use by operational personnel. One part of the 
ASRS report form focuses on the involvement and behavior of the protagonists in the “story” 
reported. In fact, the ASRS report form (NASA ARC 227B - January 1994) encourages 
reporters to describe the “Chain of Events” (i.e., how the problem arose - contributing 
factors - how it was discovered - corrective actions) as well as “Human Performance 
Considerations” (i.e., perception, judgment, decisions; actions or inactions; factors affecting 
the quality of human performance). Nearly 1 15,000 incident reports have been codified with 
this taxonomic structure and are available for further evaluation in the ASRS database. 

• The X-Form is another template that was designed to update the codification of ASRS 
reports. It was designed by ASRS personnel after several years of experience entering the 
reports into the database and conducting retrospective searches. It contains more fields 
(attributes) than the ASRS codification, which were intended to improve the descriptions of 
human-factors issues, but it has never been implemented for routine operational use by the 
ASRS. 

• The Cinq-Demi methodology was developed during the 1980s as a tool for analyzing 
aeronautical-incident reports from a human-factor’s point of view. (A brief description of 
this methodology is provided in Appendix B.) This methodology involves a structured 
analysis that focuses on identifying conditions having a high probability of leading to human 
errors. A codification form for ASRS reports was designed from the perspective of this 
methodology. Small sub-sets of ASRS reports have been codified using this methodology 
and are available for further evaluation. 

Appendix C contains a description of the study that was made of these three codification schemes. 
The comparison of their structures and a mapping of all of the parameters used in the three forms 
were the bases of the following discussion and the definition of the full set of parameters that is 
provided in Appendix D. 


5 By taxonomic structure, we mean a structured set of terms that describe some domain or topic. The taxonomic 
structure provides a skeletal structure for a knowledge base. 

6 As examples, O’Leary et at. (2002) gives examples of the type of parameters used in the British Airways Safety 
Information System (BASIS) while Murayama et al. (2002) shows some of the Performance Shaping Factors used in a 
marine incident reporting system. 
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4.1 High-Level Structure of the Three Codifications 

As explained in Appendix C, the structures of the three codification forms for ASRS reports differ in 
specificities, but their global organizations are quite similar. The information contained in all three 
can be classified into the following five categories: 

1. Time and Setting: We group in this category all the information related to the framework of the 
story (when, where. . .) and to the fixed entities (facilities. . .). 

2. Cast of entities: This category contains information on the persons and all the entities that 
evolve and take actions in order to create the story. 

3. Anomaly: This category pertains to all the information that explains why the “anomalous state” 
is anomalous. 

4. Transitions: This category is all the information that characterizes a transition in the incident 
model of states and transitions. 

5. Other: This category includes any information that cannot be classified into any of the other 
four categories. 

These categories are used only for the codification of a report and not for an in-depth analysis of the 
incident. They include neither a precise description of the sequence of transitions, nor an accurate 
time reference. The value of a well-designed codification is, primarily, in its ability to effect an 
efficient retrospective search of the database so as to produce a minimum of false positives. 
Nevertheless, worthwhile analyses can be conducted based solely on the information contained in 
the fixed fields of the form. Figure 4 consolidates the information contained in the three codification 
forms and shows the overall structure of knowledge of an incident report. 
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Figure 4. Consolidated structure of ASRS codifications. 

4.2 A “Full and Complete” Set of Parameters 

As highlighted in Wiegmann and Thaden (2003), most reports are highly informative about what 
happened, but give much less definitive information about why an incident happened. Consequently, 
there is merit to designing the clustering process so that the first level of automated filtering clusters 
incident reports on the basis of similarities in what happened. Furthermore, most of the information 
about what happened is contained in the parameters (attributes and values of the fixed fields) of the 
consolidated structure of ASRS codification (see the previous section and Appendix C). 
Consequently, our first objective was to come to agreement on the “full and complete” set of 
parameters that describe our world. Appendix D discusses the mapping of the parameters of the 
three codification forms; we assume that merging these parameters results in a nearly “full and 
complete” set of parameters that describe our world. Also, for every parameter used, we can state if 
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the concept captured is objective or not. We use the following definition: a concept is objective if its 
definition can be based on observable data. All the concepts that are not objective are called 
subjective. 1 

Our hypothesis is that a full and complete set of objective parameters of an incident in “our” world 
adequately describes what happened. We further assume that, given a full and complete set of the 
objective parameters, automated tools could reliably cluster incident reports on the basis of what 
happened and could, thereby, provide an adequate description of what happened. (The experiments 
to demonstrate this are discussed later in this report.) In fact, clustering on the what may prove to be 
pragmatically sufficient for an effective retrospective search for similar incidents, even though the 
explanation of why may be quite different for the clusters of incidents. The understanding of why the 
incident happened will rely on subjective parameters and exploitation of the free text. The proposed 
process to achieve this in a second stage of automated filtering is discussed later in Section 9 of this 
report. However, first we need to introduce the notion of a scenario and explain how it is used to 
determine similarity of reports, based on what happened. 


5 THE SCENARIO 


We use the term “scenario” in the same sense as it is often used in literature or cinematography. We 
will show how it can be used to define a clustering methodology. 

Scenario: “A summary of the plot and characters of a play or film” (Collins Dictionary) 

Let us begin with an example from literature: “Romeo and Juliet.” The scenario of this play could 
be: ‘Two young people love each other, but sociological difficulties (hate between the respective 
families) thwart their love. They are desperate when they realize the conflict between their families 
is insoluble, and their despair brings them to their deaths .” A scenario provides a global 
understanding of a story and often emphasizes the starting and ending points of the story. Several 
scenarios, more or less detailed, can be given for the same story, and some different stories (e.g., not 
in the same place, not with the same people. . .) can have the same scenario. 

An aviation incident report can be seen as a story and, as with a play or a movie, we can try to 
extract from it a scenario. An aviation incident report is the story of the evolution of our world from 
a safe state, through a sequence of events and states to a compromised or an anomalous state. If we 
return to our example (ACN 81075) in Section 3.2, its scenario could be: “A transport category 
aircraft is on a visual approach to an airport with active parallel runways. ATC changes the 
aircraft’s landing clearance at low altitude and a conflict develops with a taxiing aircraft. The 
aircraft makes a go-around maneuver .” 

The scenario notion can be useful to guiding the extraction of clusters from large databases. Indeed, 
when the search is not related to a pre-defined specific issue or a pre-selected example report, a 


7 Some of the objective factors may come from the fixed fields of a new codification form while others may have to be 
extracted from the narrative, but that is not important to this discussion. 
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meaningful way to build clusters is to group reports with similar scenarios. Identifying the main 
scenarios in a database should help experts identify major safety issues and obtain clues for 
designing intervention strategies. 

Therefore, we need to have a more accurate definition of the scenario notion. As with a play, the 
scenario highlights some parts of the story. So the specificities of the story have to be used to define 
the scenario concept. As shown by the state/transition model of an incident, an incident report has a 
specific structure that has to relate to the design of the scenario. Let us recall some of the 
characteristics of an incident: 

• It starts with a safe state, 

• The first transition leads the world into a compromised (or anomalous) state, 

• One of the transitions leads the world into an anomalous state, 

• One subsequent transition recovers a safe state. 


Latent and distal 
events and factors 


The Context 


v X / 7 

Behaviors and actions 



Years, Weeks, 
Months, Hours ago 


Hours, Minutes ago Minutes, Seconds ago 


Transitions 


► now 


Safe State Compromised State Anomalous State 


SAFE 


Figure 5. The framework of the scenario concept. 


Figure 5 is another representation of an aviation incident that is obviously similar to figure 4. 
However, we use this representation to make several important points. The initial safe state may 
extend over a long period of time and may entail latent and distal events and factors, such as 
organizational culture, laws and regulations, company policies, and education and training, that may 
influence the behavior of the protagonists. In the time frame of minutes to hours, there may be 
proximal events or factors such as weather, visibility, traffic, fatigue, and equipment that may 
influence the protagonists’ behavior and the transitions from the safe to a compromised state. In the 
time frame of seconds to minutes, the safety of the system may rely upon immediate factors of 
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communications between the pilot and the air-traffic controller. The point is that human behavior is 
context-dependent. As we seek to understand the causal factors of human error and the why of the 
incident, that context extends across all three states (safe, compromised, anomalous). Further, the 
factors of the context may evolve during the course of the incident and may even be influenced by 
the actions taken (Woods et al. 1994). 

Figures 4 and 5 may well be the representations needed to achieve our objectives. However, they are 
much too complex to be used as a basis for identifying similarities with automated clustering tools, 
at least for this initial study. Therefore, based on our experience with ASRS reports, we chose to 
emphasize three parts of the generic incident model represented in figure 4 for our concept of 
scenario: the first (safe) state (the beginning of the story); the sequence of states and transitions that 
lead the world to the anomalous state; and the anomalous state. Thus, the high-level definition of the 
scenario that has been adopted for this study is: 

SCENARIO = {CONTEXT + BEHAVIOR OUTCOME} 

With this simplified definition of scenario: 

• The Context fits the exact description of the situation in the last safe state. 

• The Behavior contains all the problematic events that occur during the transition from the 
last safe state to the anomalous state. 

• The Outcome describes why the anomalous state is considered as anomalous. It does not 
necessarily contain all the parameters used to describe the state. 

This simplified model may not apply to all worlds and, even for our aviation world, other 
representations could be stated and will, perhaps, have to be explored in future work. As an example, 
we could highlight the recovery action and the final safe state in order to study which parameters 
influenced the recovery process (and so prevented an accident). Also, this model does not provide 
for the possible changes in context across the three states, as the context of the last safe state is 
assumed to prevail throughout the incident. Nevertheless, there is merit in starting with the simplest 
possible model. 

The objectives of the first stage of automated analysis are: (1) to identify and describe the scenario 
of an incident report, and (2) to identify similar scenarios from a large database of incident reports. 
Both of these objectives benefit from our simplified definition of scenario. 

Figure 6 shows the associations between our definition of scenario and the incident model. Figure 7 
shows its relationship to the high-level structure and informational categories discussed in Section 4. 
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Figure 6. The relation of scenario to the incident model. 


We need to focus and guide the automated analyses of textual reports in order to maximize the 
chances of success with current capabilities. One way is to use domain knowledge to minimize the 
extent of the world that the automated tools must consider for each incident report, by aggregating 
the reports in the database. Accordingly, we use “phases of flight” as an aggregating principle 
because aviation-incident scenarios are likely to be identifiable with phases of flight. We may 
choose to further constrain the domain by aggregating reports by size or type of aircraft, for 
example, or by visual versus instrument flight rules, but such aggregations may well be determined 
by the purpose of the search. Next, we again use domain knowledge to identify the possible 
anomalous or compromised states for each phase of flight and, possibly, for each aircraft type. (For 
example, a near-midair collision is not an anomalous state for the taxi-out phase of flight.) Then we 
identify the subset of the “full and complete” set of parameters (objective and subjective) that could 
be relevant to, and that could define the states of, any incident in each of these phases of flight. 

The specificity of the parameters used to define the scenario’s three parts will determine the degree 
of discrimination that the automated clustering process can achieve: the parameters used to define 
the Context, Behavior, and Outcome will have to be more detailed and precise if we are interested in 
distinguishing small differences among incidents than if we are only looking for general categories. 
Consequently, the scenario is defined by the subsets of parameters that describe the Context, the 
Behavior, and the Outcome of the incident model that are specific to the “story” of a particular 
incident report. Thus, not all the parameters used to define a state of the world will be used to define 
the Context, because not all the parameters of the Context of the initial safe state are important to, or 
are causal factors of, the Behavior or the Outcome of that “story”. In the same sense, the Outcome is 
described only by those parameters that distinguish this state as an anomalous one. 

Therefore, now we have the beginning of a taxonomic structure for the first stage of clustering on 
what happened. We start with a prescribed subset of incident reports (aggregated, for example, by 
the phase of flight and by aircraft type), then identify with this subset all of the possible anomalous 
states, together with the objective descriptors that could possibly be associated with that subset of 
Outcomes. So, within each aggregation by, for example, phase of flight and aircraft type, we focus 
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on the subset of reports related to each of the identified anomalous outcomes. Finally, we identify all 
of the other objective and subjective factors that could be relevant to any incident in that sub-subset. 
This process is intended to minimize the domain that the automated system must consider in this 
first stage of filtering, and to maximize the information known about the incident. In this first stage 
of filtering, we have clustered on similarities of the objective parameters that define the Context of 
the Scenario, together with the similarities of the objective parameters that define each possible 
anomalous state associated with the Outcome of the Scenario in the aggregated subset of reports. 
(The experiment described in Section 8 was designed to explore the potential of this first filtering 
stage on the correlation between the parameters of the Context and each anomalous state.) It is 
desirable to complete this process before we attempt the next stage of automated analysis of the free 
text for its implicit (subjective) information about the clusters identified. 

An assumption previously stated is that we have adequately defined what happened (and, possibly, a 
bit of the “how ”) by identifying all of the objective parameters of the Context and the Outcome that 
existed in a particular scenario. That was the primary motivation for generating the complete list of 
the objective parameters. However, all of this, so far, is preliminary to our objective of automatically 
defining the why. In the second stage of filtering, we will see if we can isolate the subset of objective 
parameters of the Context that correlate to Behavior in that Scenario and hence to the causal factors 
of the Outcome. This process is described in Section 9. 
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Figure 7. Relations between the scenario and the categories of descriptors. 


6 PRELIMINARY CONCLUSIONS 


We have shown how we defined a generic structure of information (a taxonomic model) that is 
adaptable to the description of ASRS-like aviation incident reports. The resulting model of an 
aviation incident is postulated to be a sound basis for defining similarities among incidents reports. 
The notion of Scenario has been introduced as a pragmatic guide for identifying similarities based 
on the objective parameters that define the Context and the Outcome of a Scenario. 
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We have assumed that it is possible to design a clustering process guided by the structure of the 
scenario, and that the results will be easier to understand by an aviation expert. We now have the 
simplified structure of the Scenario. We have identified the “full and complete” set of parameters 
that define the Context of the initial safe state, and the anomalous Outcome that adequately describes 
what happened. Automated tools will use the values of these parameters to identify the Scenario and 
to cluster similar Scenarios from the ASRS database. The potential of this approach is demonstrated 
in the experiments that are described in Sections 7 and 8. 


7 A CASE STUDY 


All of the work reported in Sections 3 through 6 was preparatory to implementing an approach to 
automated clustering that is based on statistical analysis and that can be used on very large databases 
of textual reports. In this section, we describe an experiment with a different clustering technique 
that can be used only on small sets of reports, but that enables us to evaluate the model we have 
proposed. The clustering tool is described in Section 7.1. It entails a methodology based on Formal 
Concept Analysis (Ganter and Wille 1999) in which it is possible to maintain throughout the 
clustering process an explicit description of the similarities among the parameters of the reports. 

This capability will be used to check whether the similarities highlighted by the clustering process 
are valid and useful. However, there are pragmatic computational limitations to such qualitative 
analyses, so that only small sets of data (reports times parameters) can be considered. The 
statistically based tools will be used to study larger sets of reports, but they may hide the meanings 
of the similarities in a given cluster. As we want first to evaluate the validity of the model, we chose 
to start with a limited experiment based on an explicit handling of similarities. 

As explained previously, we assume that a “full and complete” set of objective parameters 
adequately describe what happened. In this study, we are going to extract from the set of objective 
parameters identified previously and in Appendix D, a description of the what of the incident in two 
parts; namely, the “Context” and the “Outcome” of the Scenario as defined in Section 5. Then a set 
of 40 ASRS reports will be codified in a formal language adapted to the clustering tool chosen (in 
Section 7.2). We use a two-step clustering process. First, we identify the Outcomes (in Section 
7.3.1), and then we analyze the Contexts that are typically associated with each of the Outcomes (in 
Section 7.3.2). In Section 7.4, we describe a limited study on the why that relied upon the Cinq-Demi 
codification. 

Before we present our study of these 40 reports and the results, we will describe how these particular 
reports were selected from the ASRS database, as well as the clustering tool that we used in the 
study. 


7,1 The Sample and The Clustering Tool 

Describing objects by their set of properties is a natural process used in several domains. A 
“concept” of the domain can be described by a set of objects: the extent, which is a collection of 
examples of elements that belong to the concept; and the intent, which is the set of their shared 
properties. For instance, the concept of airplane could be captured by a set of objects such as (B737, 
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B747, B777, A320. . .) and the set of their common properties (wings, tail, engines, pilot. . Given a 
set of objects and the set of properties associated with each object, one can automatically extract all 
the concepts involved. This is the aim of Formal Concept Analysis (Ganter and Wille 1999) based 
on well defined mathematical foundations. An extension of the FCA formalism called Generalized 
Formal concept Analysis (GFA) that enables objects to be described by a structured set of properties 
has been proposed by Chaudron and Maille (Chaudron and Maille 2000), and a tool named Kontex 
has been developed to identify concepts. Given a set of objects and their properties, the Kontex tool 
calculates all the possible concepts, shows all the relations between the concepts (generalization, 
specialization) and describes their similarities and differences by using an adapted graphical 
interface. That tool is well suited to our experiment as each incident report is directly characterized 
by a structured set of properties (as described by the taxonomy) and so can be considered as an 
object of the GFA methodology. The GFA methodology was chosen to conduct the analysis of a 
sub-set of ASRS reports. 

An important step in this experiment was selection of the set of reports to analyze. As the purpose 
was to use the explicit description of the similarities and differences between clusters developed 
with the Kontex tool, the number of reports had to be limited both as a matter of practicality in 
reference to calculation complexity, and for ease in interpreting the results. Based on a previous 
study (Maille 2002), a set of 40 ASRS reports seemed to be a practical number. Moreover, as we 
wanted to evaluate whether the taxonomy and the Scenario components supported a meaningful 
clustering on what happened, we wanted to have in the set of reports a small number of different 
Contexts and Outcomes in order to have significant clusters. Indeed, if only 40 reports are taken 
randomly from the database, they will certainly deal with a large variety of different kinds of what 
and it will be difficult to utilize the capabilities of the methodology. Therefore, we decided to focus 
on reports from the particular experiment that was concerned with “In-close Approach Changes 
(ICAC)” (Lecomte et al., informal communication)*. All of these reports deal with aircraft in the 
approach flight phase, and so the number of possible anomalies (or Outcomes) is limited. The 
clustering process would try to reveal finer similarities between groups of reports and possible links 
between Outcomes (or anomalies) and Contexts. In the ICAC experiment, around 200 ASRS reports 
had been codified by the Cinq-Demi team and 100 reports had been fully analyzed using the Cinq- 
Demi methodology of Appendix B. We chose for this experiment the first 40 reports in the Cinq- 
Demi database that had received a full analysis. 

7.2 Formal Codification of the 40 Reports 

For the 40 reports, we had both the ASRS and the Cinq-Demi codification available, but not the 
codification with the complete set of parameters described in Appendix D. We decided to rely on the 
ASRS codification as much as possible (flight phase, anomaly. . .), even if the codification is not 
exactly the one defined in our taxonomy. Then we added to this codification some parameters to 
describe the traffic, the airport (configuration of the active runways. . .) because these important 
parameters are not addressed in the ASRS codification. For these new parameters, we used the 
taxonomic structure of Appendix D. All these properties that describe the Context and the Outcome 
of the report are formally captured in a first-order language that is used by the GFA tool. 


He .... 

Lecomte, P.; Wanner, J. C.; and Wanner, N. Late runway changes. Unpublished informal communication, 2002. 
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As the Cinq-Demi codification was also available, we decided to incorporate the “error-factor” (part 
of the GOOF grid) in our formal codification. That parameter does not belong to the what , but is part 
of the description of the why. It was not the primary subject of this experiment, but will be exploited 
in Section 7.4. 

7.3 Clustering on the What 

The what is described through various objective parameters that belong to the “Context” and the 
“Outcome” parts of the Scenario model. Our clustering process starts with the identification of the 
Outcomes involved in the selected set of 40 ICAC reports. As expected, there are only a few 
different ones as they were all associated with In-Close Approach Changes. Then for each group of 
reports associated with a particular Outcome, an analysis of the related Context is conducted. 

7.3.1 First Step : Clustering on the Outcome 

As stated before, we used the ASRS taxonomy of anomalies as the codification of the Outcome. 

That taxonomy contains around 60 terms grouped into 13 categories (Aircraft Equipment Problem, 
Airspace Violation, Altitude deviation. . .). Each of the 40 ASRS reports was identified with one or 
more of these Outcomes. With the Kontex tool, the 40 reports were clustered according to their 
Outcomes. Reports were clustered by each anomaly, and also by each and every combination of the 
anomalies that had been identified within the set of 40 reports. This resulted in 50 concepts 
(clusters); a top-down analysis of these concepts was conducted. 8 

The top-down analysis identified “significant” clusters. A cluster was considered significant if (1) it 
had none or few reports that were shared with other significant clusters, (2) contained a large 
percentage of all of the reports, and (3) collectively, the significant clusters contain nearly all of the 
reports. 

This process highlighted four main groups of reports based on four anomalies: “Track or Heading 
Deviation,” “Airborne”, “Ground,” and “Near Mid-Air Collision (NMAC).” We point out that the 
three anomalies “Airborne,” “Ground,” and “NMAC” belong to the category called “Conflict” in the 
ASRS taxonomy of anomalies. So, we are able to state that we have identified two main Outcomes 
in the Scenarios of these 40 reports; namely, reports that deal with a spatial deviation (track or 
heading) and reports dealing with a conflict. This first stage of analysis also showed that the ASRS 
anomaly called “Non- Adherence to a Clearance” was often encountered, but it seemed to be a 
shaping factor in the four main groups identified. 

In addition, a set of five reports contained all the reports that were not related to any anomaly of the 
taxonomy 9 and two exceptions were identified. One exception was linked to an “Aircraft Equipment 
Problem” anomaly and the other one to an “In-Flight Encounter” anomaly. The following table 
summarizes the results. Of the five reports that do not belong to any of the four identified anomaly 


8 

Readers interested in how such a top-down analysis is performed can refer to the report by Maille (Maille 2002) where 
a similar analysis is performed for 44 ASRS incident reports. 

9 Some reports in the ASRS database have no identified anomalies but have been entered into the ASRS database 
because the ASRS experts considered the sequence of events to be interesting from a safety point of view. 
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categories, four reports are linked only to a problem of non-adherence to a procedure (i.e., a FAR or 
a clearance) and one report relates to an altitude deviation. 


TABLE 7.1. CLUSTERS OF REPORTS TO ANOMALIES 
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7.3.2 Second step: Clustering on the Context 

In the first step, we clustered the 40 reports by their Outcomes (the ASRS-defined anomalies). In the 
next step, we wanted to cluster on the shared Context within each significant cluster. This Context 
was fully described by the objective parameters as they were provided in the codification of each 
ASRS report. Therefore, we used the Kontex tool once again to cluster on each objective parameter 
and all combinations of objective parameters within a significant cluster of Outcomes. The idea was 
to explore all the possible concepts based on the formal codification of both the Outcome and the 
Context, and to determine whether a specific Context could be associated with each of the four 
particular Outcomes highlighted in the first step. The main results of this second stage of clustering 
and the top-down analysis are summarized here. 

“Track and Heading.” The Context shared by the 12 reports identified with this anomaly is: “An 
aircraft is in the approach flight phase to an open and controlled airport. The aircraft is controlled at 
the beginning of the incident by the TRACON.” In addition, in most cases (in 9 of the 12 reports), 
the aircraft was in Class B airspace and parallel runways were active. 

“Airborne.” The Context shared by the 10 reports in this category is: “Two aircraft are in the 
vicinity of an open and controlled airport. One of them is in the approach flight phase.” This shared 
Context by itself was not adequately discriminating, but the analyses showed an interesting group of 
6 reports that shared the following properties: “The two aircraft are in the same phase of flight 
(approach) in the Class B airspace. There is some traffic and parallel runways are active.” The four 
other reports shared only the property of being controlled by the tower. 

“Ground.” The Context shared by the 4 reports in this category is: “An aircraft is in the approach 
flight phase to an open and controlled airport. The aircraft is controlled by the Tower.” Then further 
analysis shows that the more common situation (3 reports) is that “there is another aircraft and some 
traffic.” The only report without another aircraft in the Context deals with a conflict between the 
first aircraft and airport workers on the runway. 
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“NMAC.” The Context shared by the 3 reports identified with this anomaly is: “Two aircraft are in 
the approach flight phase to an open and controlled airport. Parallel runways are active and there is 
some traffic. One aircraft is a Medium Large Transport (MLT) and the conflict develops while they 
fly in Class B and D airspace.” 

At first glance, the four Contexts appear to be similar. Let us highlight their similarities and 
differences. They all deal with “an aircraft in the approach flight phase to an open and controlled 
airport.” This is not a surprise, as it is a direct consequence of our selection of this particular subset 
of 40 reports. Therefore the differences in Contexts rely on a finer level of granularity in the 
descriptions of the Contexts. 

We point out, for example, that the three Contexts associated with conflicts (i.e.. Airborne, Ground, 
and NMAC) contain the descriptors “traffic” and “2 aircraft” but not the descriptor “TRACON.” 
In contrast, the “Track and Heading” deviation category seems not to be directly influenced by the 
traffic or the presence of another aircraft, but generally starts to develop while the aircraft is still 
under the control of the TRACON (coordination during the transition of control from the TRACON 
to the Tower could be a source of Track and Heading deviations). Conflicts, on the other hand, are, 
as expected, directly related to a problem of traffic and to the simultaneous presence of two aircraft 
around the airport. 

The differences in the Contexts associated with the three conflicts are more subtle. First the NMAC 
Context is a special case of the Airborne Context (2 aircraft in the same flight phase, traffic, parallel 
runways, and class B airspace). This is an interesting result as we can also point out that the NMAC 
anomaly is a special case of airborne conflict. What makes the difference between the two contexts 
is that another airspace, D, is also involved in our NMAC Context and one aircraft is a MLT. Thus 
the differences between these two contexts are small, but the anomalies are also quite similar. It will 
require a larger set of reports and more detailed descriptions of the Contexts (more objective 
parameters) to give more reliable conclusions about the differences in what happened in these two 
cases. 

We will now focus on the differences that automated clustering reveals in the Contexts associated 
with Ground and Airborne anomalies. The Ground context contains the descriptor tower , but neither 
the same flight phase for the 2 aircraft, nor the class B airspace, nor the parallel runways. Thus, for 
these 40 reports, airborne conflicts generally entail the presence of two aircraft in the same flight 
phase to an airport where parallel runways are active. In contrast, Ground conflicts develop only 
under tower control and are not directly linked to parallel-runway configuration, just as one would 
expect. 

7.4 An Attempt to Capture the Why 

As stated previously in this report, we have introduced into the formal codification one part of the 
Cinq-Demi taxonomy, which captures some conditions conducive to human error. This part related 
to understanding the Behavior of the Scenario was not a primary aim of this small case study, but as 
the Cinq-Demi codification was available, it required only a little additional work to see if there 
were interesting results. 
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The study of these added parameters indicated that nearly all the conflicts were associated with a 
misrepresentation of risk by the air traffic controller. (The concept of misrepresentation is first 
introduced by Cinq-Demi as presented in Appendix B and is discussed further in Section 9 and 
Appendix E in this report.) Thus the behavior of the controller seemed to greatly influence the 
possible occurrence of a conflict. 

The reports dealing with track and heading deviations seemed to be split into two clusters, 
depending on whether or not the flight crew had a correct representation of the trajectory of the 
aircraft. Thus either the behavior of the flight crew or that of the air traffic controller was a primary 
factor in the deviation. Moreover, for nearly all of the track and heading deviations, workload was a 
concern, while it seemed to be a marginal parameter in the case of conflicts. 

7.5 Conclusions of the Case Study 

This limited experiment showed that, within the phase of flight selected for the Context of these 40 
reports, a first clustering process based on the description of the Outcome generated well- separated 
groups of reports. Then, the analysis of the related Contexts was able to point out discriminating 
parameters (e.g., active parallel runways and aircraft in the same phase of flight were related to 
airborne conflicts, while the transition from the TRACON control to the tower control was more 
relevant to track and heading deviations). Of course, with a so small a number of reports, one should 
be careful about the reliability and the generalization of the results. The purpose of this study was to 
test the value of the model, and not so much to come to conclusions about the links between 
Contexts and Outcomes in this small set of reports. 

Moreover, the rough codification of the why shows that misrepresentation seems to be a common 
factor in all four of the anomalous outcomes of ICAC, and that some subjective parameters (for 
instance, workload) can be a contributing factor to some anomalies (or to some contexts). 
Furthermore, the indications are that the major clustering criteria for Behavior could be the person 
who had an anomalous behavior. 


8 CORRELATIONS BETWEEN OUTCOMES AND CONTEXTS 


8.1 Introduction 

The Scenario model is concerned with the relationships among the Contextual factors of the last safe 
state. Behaviors, and anomalous Outcomes of a safety incident. We decided to examine ASRS data 
for evidence of such relationships. 10 In the fixed fields of the coded forms, ASRS database records 
contain a good deal of structured information relating to the Context and Outcomes of reported 
safety events, but very little structured information relating to the Behaviors of the people and 
automation that contributed to the events. Therefore, we limited this initial examination of ASRS 


10 This section of this research report summarizes work done by a team of researchers from Battelle Pacific Northwest 
Division. The team, under the leadership of Dr. Thomas Ferryman, included Ms. Amanda White, Dr. Christian Posse, 
and Ms. Andrea Swickard. 
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data to the relationship between the context of reported incidents and their outcomes, exclusively. 

Our intent was to extract behavioral data from ASRS narratives 11 and incorporate those data into an 

expanded future analysis if this initial investigation proved fruitful. 

8.2 Goals 

The goals of this investigation were to determine whether: 

1. There are statistically significant relationships in ASRS incident reports between coded 
Contextual Factors, on the one hand, and coded anomalous Outcomes, on the other. 

2. Any such statistical relationships that are observed in ASRS data are amenable to operational 
interpretation by subject-matter experts. 

8.3 Approach 

During this research effort we: 

1. Created a structured analysis table from the ASRS data. 12 

2. Examined the statistical relationships between contextual factors and anomalies using the (a) 
classification and regression tree (CART) method, and (b) cross-tabulation analysis. 

3. Clustered the contextual factors found in the ASRS data into groups based (roughly) on their 
frequency/infrequency of co-occurrence. These groups can be thought of as recurring Contextual 
Patterns. 

4. Examined the statistical relationships between Contextual Patterns and anomalous Outcomes 
using cross-tabulation analysis. 

5. Developed graphical depictions of the results of both CART and cross-tabulation analyses to aid 
their interpretation. 

6. Presented the findings to subject-matter experts (SMEs) to determine whether the patterns 
revealed by the statistical analyses were operationally plausible. 

This section of the current report summarizes the approach and findings of that investigation. 

8. 3. 1 Data/T axonomies 

Data for this research were obtained from the ASRS database. The active ASRS database contained 

109,225 records within the scope at the time the data request was made. 


1 1 ASRS report narratives are a rich source of information regarding the behaviors of pilots, air traffic controllers, other 
persons, and automated agents during the course of safety events. However, the unstructured nature of these data 
creates an analytical challenge. 

19 

Each row in the table corresponded to a reported ASRS incident. Columns described the contextual factors present 
during the reported event and an anomalous outcome that resulted. Since any given ASRS report may describe more 
than one anomaly, some reported events appear more than once in the analysis table. 
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Scope of Database 

The scope of the analysis was limited to ASRS incidents involving at least one air transport on a 
passenger or freight mission flying under Part 121 rules. 

Contextual Factors 

As noted previously in Section 4 and Appendix C, many contextual factors relevant to safety 
incidents are encoded in ASRS fixed fields. The coded Contextual Factors used in this study fall into 
the following categories: 

• Time including year, month, day of week, and quarter of day 

• Place including altitude, location (airport, intersection, etc.) and involved ATC facilities 

• Physical environment including flight conditions (VMC/IMC), ceiling, visibility, and light 
conditions (dawn, day, dusk, night) 

• Aircraft characteristics including make-model (implicitly, weight, number of engines, etc.), 
mission, navigational-method-in-use, and flight phase 

• Hazardous situational factors including problematic airport configurations, airspace 
designs, departure/approach procedures, navigational aid configurations, and ATC/airport 
procedures. 

We used 257 ASRS contextual factor codes that fall into the above categories. 13 This set of fixed 
fields was supplemented with 267 context-related words extracted from the report narratives. CART 
analysis can accommodate large numbers of potential explanatory variables. However, cross- 
tabulation analyses lose statistical significance when cell sizes become too small. Thus, for the 
purposes of the cross-tabulation analysis, it was necessary to cluster ASRS contextual factors into 
Contextual Patterns (groups of Contextual Factors evidenced by very frequent or very infrequent 
co-occurrence). 

We arrived at these contextual patterns using standard clustering methods. The raw data evidenced 
2,882 distinct Contextual Factor sets (excluding location identifiers). A hierarchical clustering 
method was used to group these into ten Contextual Patterns. These are broadly described in 
table 8-1. Each ASRS report was associated with one of these ten Contextual Patterns based on its 
proximity to Pattern centroids. 


13 


We did not treat individual locations as distinct contextual factors. If we had, the number of such factors would have 
been measured in the thousands. 
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TABLE 8-1. KEY CHARACTERISTICS OF THE 10 DOMINANT CONTEXTUAL 
PATTERNS OBSERVED IN THE ASRS DATA SET 


Pattern 

Contextual Factors 
Unusually Present 
in Pattern Members 

Contextual Factors 
Rarely Present 
in Pattern Members 

1 

Climb phase 

Thunderstorm; military/small aircraft 

2 

Adverse weather 

Military aircraft, special purpose aircraft, 
ultralights; 

3 

Military aircraft, special-purpose 
aircraft, ultralights; adverse weather 


4 

Descent phase 

Small aircraft; adverse weather 

5 

Military fighters and trainers; mid- 
size transports 

Adverse weather (except 
thunderstorms) 

6 

Ground phase 

Adverse weather 

7 

Precipitation and obscuration 
factors; military aircraft 

Cruise phase 

8 


Military aircraft; adverse weather factors 

9 

Landing phase 

Military aircraft, special purpose-aircraft, 
ultralights; adverse weather 

10 

Military aircraft, special-purpose 
aircraft, ultralights 

Adverse weather 


Outcome Categories 

We selected ten anomalous Outcome categories from among the anomalies defined in Appendix D 
for our analysis. These categories were selected by subject-matter experts. The chosen Outcomes 
were easily mapped to ASRS database codes. Table 8-2 shows the chosen Outcomes and the 
corresponding ASRS Anomaly codes. Any given aviation safety incident may involve more than one 
adverse Outcome. For example, many ASRS incident reports that involve Outcomes 1 through 8 (or 
10) also fall under Outcome 9, Non- Adherence to Rules. Thus, some reported incidents appear more 
than once in the analysis data set. 
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TABLE 8-2. ANOMALOUS OUTCOME CATEGORIES USED IN THE STUDY 14 


Outcome Category 

Relevant ASRS Anomaly Codes 

# of 

Reports 

Set 

Label 

1 

Aircraft Equipment 
Problems 

Aircraft_Equipment_Problem. Critical 
Aircraft_Equipment_Problem.Less_Severe 

21,802 

2 

Altitude Deviation 

Altitude_Deviation. Overshoot 
Altitude_Deviation. Undershoot 

8,018 

3 

Airborne Conflict 

Conflict.Airborne_Critical 
Conflict.Airborne_Less_Severe 
Conflict. NMAC 

14,427 

4 

Ground Conflict 

Conflict. Ground_Critical 
Conflict. Ground_Less_Severe 

4,637 

5 

Runway Incursions 15 * 

Incursion. Runway. Other 

4,201 

6 

Landings without Clearance 

Incursion. Landing_Without_Clearance 

1,057 

7 

Inflight Weather Encounters 

lnflight_Encounter. Weather 

4,472 

8 

Maintenance Problems 

Maintenance_Problem.lmproper_Documentation 

Maintenance_Problem.lmproper_Maintenance 

2,371 

9 

Non Adherence to Rules 

Non_Adherence. Clearance 

Non_Adherence.FAR 

Non_Adherence.Published_Procedure 

47,748 

10 

Airspace Violations 

Airspace_Violation. Entry 

490 


8.4 Results 

8.4.1 CART Analyses 

Three Classification And Regression Tree (CART) analyses were performed. These analyses 
differed with respect to the contextual factors that were used as explanatory variables as follows: 

Analysis 1: Used 257 contextual factors drawn from ASRS fixed fields 

Analysis 2: Used a reduced set of 84 contextual factors drawn from ASRS fixed fields 

Analysis 3: Used 267 context-related words drawn from the ASRS report narratives. 

Analyses 1 and 2, which relied on ASRS fixed fields as the source of contextual information, 
seemed to produce the best results. One plausible explanation is that the coding in ASRS fixed 
fields, which draws on all information in the ASRS reporting form, is more consistent than 
contextual references in report narratives. Table 8-3 provides summary output for Analysis 2 which 
yielded the most useful CART results. 


14 There were 109,225 reports in total in the ASRS data set from which these observations were drawn. 

15 The ASRS defines a runway incursion to include any use of a runway that is not authorized by ATC. Thus, ASRS 
Anomaly Code “Incursion. Runway. Other” would be more properly labeled “Runway Incursions Other Than 
Landings Without Clearance.” Landings without clearance were separately categorized for the instant analysis as 
Outcome Category Set 6. 
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TABLE 8-3. CART ANALYSIS RESULTS SHOWING CONNECTIONS AMONG 
84 CONTEXTUAL FACTORS CODED IN ASRS FIXED-FIELDS AND 
ANOMALOUS OUTCOMES 


Outcome Category 

Contextual Factors with 
the Most Statistical Explanatory Power 

Set 

Label 

1 

Aircraft Equipment 
Problems 

Mission is passenger, flight phase is climbouHntermediate altitude, 
climbout-takeoff, other-emergency, other-divert, cruise-level, 
cruise-other, ground-preflight or ground-parked, may involve a low- 
wing aircraft or widebody transport, mission is passenger 

2 

Altitude Deviation 

Flight phase is climbouHntermediate altitude or descent-other or 
descent-intermediate altitude 

3 

Airborne Conflict 

Two or more crews involved, flight phase is cruise-level, cruise-other, 
climbout— initial, climbout-takeoff, descent-approach, two or more 
aircraft involved, low-wing aircraft 

4 

Ground Conflict 

Two or more crews involved and flight phase is ground-other, 
ground-holding, ground-preflight, ground-taxi, ground-parked, 
climbout-takeoff, landing-roll or landing-other, aircraft may be 
widebody or medium large transport 

5 

Runway Incursions 
(other) 16 * 

Two or more crews involved, flight phase is ground-other, ground-taxi, 
ground-hold, ground-position and hold 

6 

Landings without 
Clearance 

Flight phase is landing-other, landing-roll or descent-approach, may 
involve 2 or more crews 

7 

Inflight Weather 
Encounters 

Flight phase is cruise-other, descent-approach, descent-other or 
landing-other, may involve high-wing aircraft or 2 or more crews 

8 

Maintenance 

Problems 

Flight phase is ground-maintenance, ground-parked or cruise-level, 
may involve low-wing aircraft 

9 

Non Adherence to 
Rules 

No positive evidence 

10 

Airspace Violations 

Flight phase is cruise-level, may involve low-wing aircraft 


8.4.2 Cross-Tabulation Analysis 

We cross-tabulated the data set using the ten identified Contextual Patterns as the rows and ten 
chosen anomalous Outcomes as the columns. We then computed the ratio between the observed 
number of observations in each cell and the statistically expected number of observations. Figure 9 
shows the results. It is color-coded to highlight ratios that are unexpectedly high and those that are 
unexpectedly low. 


16 The ASRS defines a runway incursion to include any use of a runway that is not authorized by ATC. Under this 
definition, an unauthorized landing is a type of runway incursion. Thus, Outcome Category 5, Runway Incursions. 
Others should be understood to mean “runway incursions other than landings without clearance.” Landings without 
clearance were separately categorized as Outcome 6 for the purposes of this study. 
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Figure 8. Cross-tabulation results showing ratio of observed over expected cell frequencies. 


Figure 8 clearly reveals strong statistical relationships between the Contextual Patterns and 
anomalous Outcomes described in ASRS incident reports. For example, Context 2 (roughly, adverse 
weather) is negatively correlated with Outcome 6 (landings without ATC clearance). This is 
consistent with prior ASRS research that suggested that most landings without clearance occur in 
VMC conditions. Favorable weather can contribute to flight crew complacency and associated errors 
of omission. Context 10 (roughly, the presence of military or special purpose aircraft in adverse 
weather) appears to significantly increase the statistical likelihood of Outcome 3 (airborne conflicts). 
Subject matter experts agree that airborne conflicts are more likely to arise when aircraft with very 
different performance characteristics (e.g., air transports, military aircraft, or special-purpose 
aircraft) are in the same traffic mix. This potential is heightened in marginal weather conditions. 
Similar interpretations can be made of a number of other statistical relationships observed in the 
cross-tabulation results. 

8.5 Lessons Learned 

While far from definitive, the research described in this chapter yielded some important lessons. 
First, we are encouraged to believe that relationships that are both statistically and operationally 
meaningful exist between Contextual Factors/Pattems, on the one hand, and specific types of 
unwanted aviation safety Outcomes, on the other. Second, we recognize that the multiplicity of 
contextual factors that may be present during aviation safety events creates analytical challenges 
(i.e., the dimensionality needs to be reduced through recurrent pattern identification). Third, we 
recognize the danger that studies such as these can produce analytical results that are tautologies 
(things true by definition) rather than true insights. This happens when an Outcome has contextual 
factors built into its definition (e.g., landings without clearance, by definition, occur during the 
landing phase). Fourth, we better understand the importance of bringing domain expertise into the 
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research process at the beginning of the research study rather than reserving its application to the 
interpretation of research results at the end. Domain expertise can be used to achieve dimensionality 
reduction based on operational rather than mathematical considerations, identify implicit tautologies, 
and otherwise assist study design and execution. 


9 ON THE CODIFICATION OF THE WHY 


We now come to the primary purpose of this study. 

In Section 5, we described our concept of the Scenario, 

SCENARIO = {CONTEXT + BEHAVIOR OUTCOME} 

in which we consider the Context to be that of the last safe state and the Behavior results in the 
transition to the Outcome. When the Outcome is an anomalous (unwanted or compromised) state, 
the last safe state is identified as a precursor. 

In the experiment discussed in Section 8, we clustered incidents on the basis of similar patterns of 
parametric values defining their Contexts and their Outcomes. We have adequately described what 
happened, but we have not yet identified the causal factors of the Behavior that produced the 
transition from the last safe state to the unwanted Outcome - the why. For this, we must rely on a 
second stage of clustering. We will use the results of clustering on what combined with domain 
knowledge to minimize the extent of the world that the automated tools must consider in this second 
stage of clustering. 

In order to answer the most important human-factors questions about an incident, we must extract 
causal information from the free narrative of the incident report. We need not do a perfect job of 
this. The identification of what happened in the first stage of clustering already achieves much of 
what is needed for an effective retrospective search. Furthermore, our aim is not a definitive 
explanation of why a given incident occurred. We cannot expect to automate a completely reliable 
understanding of the why. We only need the capability to expedite a search on the why, to enable an 
exploration of their commonalities, and to minimize the labor of the human expert in arriving at a 
satisfactory explanation. It is sufficient to restrict the space of possible causal factors, and, in some 
cases, to identify a set of related incidents that includes almost all those that would be selected by an 
expert as similar to the target incident. 

It is fortunate that we do not have to be perfect in automatically extracting the precise why events 
happened, because this study is based solely on ASRS incident reports. As products of a voluntary 
reporting system, the reports in the database have some inherent limitations. They cannot be viewed 
as a random sample of the population of aviation incidents, they may contain reporting biases, and 
their factual correctness cannot be verified. Moreover, retrospective experiential reports like the 
ASRS reports are not reliable reports about why incidents happened. Ericsson and Simon (1993) 
have reviewed and analyzed many decades’ worth of research on the uses of verbal reports. They 
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repeatedly emphasize two points, both of which are relevant and of concern to our experiment based 
on aviation incident reports: 

• People have no ability to verbalize their own perceptual and cognitive processes. But it is 
precisely the understanding of the perceptual and cognitive processes that is fundamental to 
identifying the causal factors of the Behavior. 

• People have little or no ability to provide any accurate information about their performance 
or cognition after a short time has passed. However, ASRS reports are usually provided 
several hours after the target events. 

Furthermore, people cannot report on the perceptual and retrieval processes that determine which 
thoughts or patterns reached their attention or why a given thought was attended. Perhaps the best 
we can hope for is a sequentially correct report of the most salient attended objects and events, 
especially those attended during impasses in normal, smooth performance. For the most part, the 
reported events will relate to the what , but when they are related to why , they are almost certain to be 
directly linked to concrete perceptual factors. While this source of our information about the why 
may be flawed, it is nevertheless the best source we have for the operator’s perspective of the 
incident, and it deserves to be mined for whatever information it contains. 

In this second stage of automated analysis, we need to rely on knowledge of human behavior to 
narrow the possibilities of the why in order to “aim” the automated system in the proper direction. In 
fact, at this stage of the research, we are willing to omit many plausible (albeit rare) causal factors of 
human behavior (such as physiological and psychomotor factors) if we can aid the analyst in the 
identification of a few important common ones. In our initial attempt to cope with this complex 
problem, we propose that the Behavior entailed in transitioning from the safe state to a compromised 
or anomalous state of the Outcome is always associated with a loss of “Situation Awareness.” 

Endsley (1988) defines Situation Awareness (SA) as a person’s “perception of the elements in the 
environment within a volume of time and space, the comprehension of their meaning, and the 
projection of their status in the near future. ” SA is a label that is often used to refer to many of the 
cognitive processes entailed in attentional dynamics, maintenance of a world model, and prediction. 
This definition fits well with our view of Behavior as we use it in the concept of Scenario. 

Moreover, the concept of associating any Behavior that results in an anomalous Outcome with a 
failure of Situation Awareness fits with our objective of identifying systemic factors conducive to 
human error. SA is concerned with the operational state of an expert human performer in a dynamic 
and potentially dangerous environment. In our aviation world, we are considering pilots and air- 
traffic controllers operating in the global civil aviation environment. Other studies of SA have 
focused on challenging military operations, such as command and control in joint-operations 
combat. Still others have studied automobile drivers, anesthesiologists, space mission ground- 
controllers, and firefighters. We base our approach on the substantial body of literature reporting on 
a variety of perspectives of SA and its role in human behavior. (See, for example, Durso and 
Gronlund 1999, Shively et al. 1997, and Sohn and Doane 2000.) 

SA is the converse of the concept of Misrepresentation that is an “Error Factor” in Cinq-Demi’s 
GOOF grid. (See Appendix B.) Cinq-Demi uses Misrepresentation to mean situations when the 
model used by the operator to understand events and act accordingly is, for any reason, not 
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consistent with the “real world.” Misrepresentation involves the reporter’s failure to update his or 
her mental model of system status; it can also involve use of a decision model that may be generally 
valid, but is too simple, or is inappropriate to the reported situation. Misrepresentation is viewed 
here as synonymous with loss of Situation Awareness. 

This notion — that the loss of SA (or Misrepresentation) always underlies the Behavior associated 
with the transition from a safe state to a compromised or anomalous state — has some justification, at 
least with regard to ASRS reports. In every study we have conducted in which the full analyses 
using the Cinq-Demi methodology have been applied to ASRS reports, we have concluded that 
Misrepresentation dominated all of the Error Factors. Further, experienced ASRS analysts agree that 
“Misrepresentation” (in its most general interpretation) is the dominant factor in, by far, the majority 
of the ASRS reports in the database, except in some cases of equipment failure. 

Hartel et al. (1991) found SA to be the leading causal factor in a review of 175 military aviation 
mishaps. Endsley (1995a) investigated the causal factors underlying aircraft accidents involving 
major air carriers from 1989 to 1992. Of 24 accidents, 17 involved human error and, of these, 15 
were associated with a failure of Situation Awareness. Endsley concluded, “This study provides 
good evidence that problems with Situation Awareness are indeed a primary factor underlying 
aviation accidents. ” This conclusion was further validated by the results of the studies reported in 
Jones and Endsley (1996) and Gibson et al. (1997), although these entailed only small subsets of 
incident reports from the ASRS database. 

Therefore, it may not be too extreme to start with the assumption that the human behavior failures of 
every incident entail loss of Situation Awareness, especially as the ASRS database will be our 
resource for experiments. This provides us with a model to guide the automated clustering. 
Moreover, the automated clustering processes to be used in this experiment lend themselves to 
testing this assumed model, as will be described later. 

However, SA by itself is non-constructive and insufficiently discriminating. To make progress with 
the analysis and measurement of our conceptual model, it is necessary to break SA down into more 
concrete and constructive components. Fortunately, we can draw on an extensive SA research 
literature to accomplish this. (See Appendix E for a discussion of this literature and research in 
related domains of human factors, skilled performance, and behavioral decision theory. Appendix E 
also discusses some of the complex processes and interactions that we are ignoring in this initial 
experiment.) 

The human-factors research community (see, for example, Endsley 2000a, Endsley 2000b, and 
Shively, R. J., et al. 1997) has identified the following sequential stages or aspects of Situation 
Awareness: 


35 



SITUATION AWARENESS 


DETECTION 


A 

RECOGNITION 

A 

INTERPRETATION 

A 

COMPREHENSION 

A 

PREDICTION 


We propose that the discriminating factors of Behavior in our model of Scenario are failures to 
Detect, Recognize, Interpret, Comprehend, or Predict. (See Appendix E for descriptions of each of 
these five components of SA.) 

There are three benefits to our experiment that can be realized by adopting this breakdown of 
Situation Awareness. First, the determinants are constructive in the sense that each might be 
identifiable with specific words or phrases in a set of reports. Second, they are constructive in the 
sense that we may be able to relate each to specific objective parameters of the Contexts and 
Outcomes of each Scenario. Third, as they are naturally sequential, they have the potential for 
augmenting the information on the sequence of events (or sub-events) during the transition from the 
last safe state to the outcome. All of these features can be useful to “tuning” the automated analyses 
of this second stage, as will be described later. 

A number of previous studies have highlighted levels or stages of SA that are closely related to our 
list of discriminating components: Detection, Recognition, Interpretation, Comprehension, and 
Prediction (DRICP). For example, Endsley developed the taxonomy in table 9-1 for classifying and 
describing errors in SA (Endsley 1994, 1995a, and 1995b). The factors affecting SA at each of the 
three levels of table 9-1 correspond to the components of SA we propose to use. Detection and 
Recognition are necessary for Level 1 SA (Endsley 1996, 2000a, and 2000b). Interpretation and 
Comprehension are necessary for Level 2 SA (Endsley 1996, 2000a, and 2000b). A person with 
Level 2 SA has been able to derive operationally relevant meaning and significance from the Level 1 
data perceived. Endsley 2000a and 2000b emphasizes that the defining role of prediction is the 
highest level (Level 3) of SA. We may well find that it is not possible to discriminate automatically 
to the five levels of detail of our taxonomy, in which case we will try to adapt our analyses to 
Endsley’ s three-level taxonomy of perception, comprehension, and projection. In any case, 

Endsley’ s lower- level descriptions of each of the three levels in table 9-1 will help us develop 
representative concepts, words, or phrases that a reporter of an incident might use to indicate the 
components of SA. 

Jones and Endsley (1996) found that experts achieved a limited degree of success in categorizing a 
small sub-set of ASRS reports at these three levels of table 9-1. They also found that the distribution 
of errors among the three SA Levels in the 143 ASRS incident reports of this study was comparable 
to that found in a previous study of 17 NTSB accident reports (Endsley 1995b). 
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TABLE 9-1. TAXONOMY OF LEVELS OF SITUATION AWARENESS 

Level 1: Fail to perceive information or misperception of information 

• Data not available 

• Hard to discriminate or detect data 

• Failure to monitor or observe data 

• Misperception of data 

• Memory loss 

Level 2: Improper integration or comprehension of information 

• Lack of or incomplete mental model 

• Use of incorrect mental model 

• Over-reliance on default values 

• Other 

Level 3: Incorrect projection of future actions of the system 

• Lack of or incomplete mental model 

• Over-projection of current trends 

• Other 


With this assumed model of Behavior, we now have a taxonomy of sequential, constructive, 
discriminating factors of Behavior that could help explain the why and how of an incident. We next 
need to identify which of these behavioral factors (i.e., failure to Detect, Recognize, Interpret, 
Comprehend, or Predict) were present in the Scenario of the subset of incident reports developed 
from the first stage of clustering on what happened. Then we need to identify the objective factors of 
the Context that are related to the identified behavioral factors. 

Accordingly, in order to maximize the potential of the automated extraction of this information, we 
expect to do the following in the experiments that we will conduct during the next year: (This 
corresponds to the second stage of clustering in the procedure diagrammed in figure 9.): 

1. An expert in human factors will work with operational experts (ASRS analysts) to develop 
representative concepts, words, or phrases that a reporter of an incident might use to indicate the 
components of SA (DRICP). Examples are presented in table 9-2. 

2. We will then use such exemplary phrases with the tool called QUORUM-Perilog (McGreevy 
and Statler 1998) to search the entire ASRS database for similar phrases. On the basis of that 
search, we will develop a set of words, phrases, and phraseologies related to each of the 
discriminating components, DRICP, of SA. 

3. With the help of operational domain experts and experts in human factors, we will develop 
subsets of the previously labeled subjective factors that relate to each of the components of SA. 
We will then use the set of phrases developed in step 2 above, together with the subset of 
subjective factors associated with each of the discriminating components of SA, to “tune” the 
automated analysis in the second stage of clustering. This second stage will cluster reports by 
similarity of the failures in SA (i.e., failure to detect and/or to recognize and/or to identify, etc.) 
that occurred from each cluster of incident reports identified in the first stage of analysis as 
similar on the basis of what happened. 
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TABLE 9-2. REPRESENTATIVE TEXTUAL EXPRESSIONS 



Concepts 

Words 

Phrases 

Lack of 
Detection 

threshold, change, 
adaptation level, 
signal quality, 
discrimination, noise 

did not notice, see, 
hear, monitor 

1 did not notice that the 
MCW light was on. 

We were not monitoring 
altitude. 

Lack of 
Recognition 

attention, familiarity, 

type/kind/category, 

importance 

misunderstood, 
misread, mis-heard, 
confused, unknown, 
novel, new, unfamiliar 

An unfamiliar annunciation 
appeared on the MCP. 
PNF mis-heard the 
clearance. 

Lack of 
Interpretation 

relations, reasoning, 
language, training, 
specialized knowledge 

incorrectly, not fully, 
incompletely; not 
realize meaning, 
importance 

CAPT did not realize how 
soon we needed to start the 
descent. 

Lack of 

Comprehension 

causality, explanation, 
diagnosis, 
intervention, FDIR 

lost track of, mistake, 
wrong, error, why, 
misunderstand 

We did not understand why 
the altitude capture failed, 
(notice in this case how 
detection, recognition, and 
interpretation are satisfied: 
‘the altitude capture failed’ is 
an interpretation of cockpit 
information that has been 
detected and recognized; 
still there can be a 
comprehension failure) 

Lack of 
Prediction 

prediction, 
preparation, 
expectation, 
prevention, avoidance 

not expect, 
unexpected, 
unforeseen, not 
remember to 

The weather had 
deteriorated at our alternate. 
We got an unexpected 
runway change. 


4. Again, an expert in human factors will work with ASRS analysts to identify which of the 
objective parameters of the Context might relate to a failure of each of the discriminating 
components of SA (i.e., which of the objective parameters might contribute to a failure to Detect, 
which to a failure to Recognize, which to a failure to Interpret, which to a failure to 
Comprehend, and which to a failure to Predict). This step is intended to guide the automated 
search of the next step. 

5. The automated analysis in Step 3 identifies which of the components of SA pertain to the 
Behavior of the common Scenario in a cluster of incident reports. Next we will automatically 
compare the objective parameters of the Context for that Scenario with the list of those 
parameters that the experts identified as relevant to each of the pertinent components of SA. This 
comparison will enable us to identify the subset of objective parameters of the Context that is 
related to each of the discriminating factors of the Behavior identified with that Scenario. 
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6. After identifying which of the components of SA are entailed in the Scenario, it might be 

possible to extract some information about the chronology (linkage) of the sub-events from the 
natural sequence of the components of SA by relating these to their corresponding existing 
contextual factors. The consequent completion of the Scenario that includes this chronology of 
the contextual factors and their relationship to the components of SA permits the postulation of 
effective intervention. 

We use the DRICP framework as though Detection, Recognition, Interpretation, Comprehension, 
and Prediction occur in sequential order, each successive stage using the output of the preceding 
stage. This simplification is both necessary and justifiable in the initial part of our work. It is 
necessary to keep the analysis tractable, and it is justifiable because there is every reason to believe 
that ASRS reports are usually delivered as sequential narratives. 

However, as Carroll et al. (2001) documents, citing Neisser (1976), human cognition is a cyclic 
process in which prediction facilitates comprehension and interpretation, and in which 
comprehensible and interpretable events are more easily detected and recognized than are 
unpredictable and incomprehensible events. In fact, Jones and Endsley (1996) point out that many 
Level 2 SA errors (for example, misinterpretation of landmarks) can be attributed to incorrect 
expectations (erroneous predictions), which then cause a persistent misrecognition and 
misinterpretation of perceptual data. 

During the next year, we expect to conduct the experiment described above, applying the paradigm 
of Situational Awareness to automated clustering on the parameters extracted from ASRS incident 
reports associated with erroneous human Behavior. We will present the results of this experiment in 
a subsequent report. 
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Figure 9. Process for second-stage analysis. 


10 SUMMARY AND PROJECTIONS 


In this study, we have defined a generic structure of information (a taxonomic model) that is 
postulated to be a sound basis for defining similarities between incidents like those described in 
ASRS-like aviation incident reports. On the basis of this structure, we have introduced the 
simplifying structure of the Scenario as a pragmatic guide for identifying similarities of what 
happened based on the objective parameters that define the Context and the Outcome of a Scenario. 

We assume that it is possible to design an automated clustering process guided by the structure of 
the Scenario, and that the results will be easy for human experts to understand. We have identified 
the “full and complete” set of parameters that define the Context of the initial safe state, and the 
anomalous Outcome. Our assumption is that this complete set of parameters adequately describes 
what happened. Automated tools will use the values of these parameters to identify the Scenario and 
to cluster similar scenarios from the ASRS database based on what happened. We have 
demonstrated the potential of this approach in the experiments described in this report. 
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The limited experiment of the “Case Study” discussed in Section 7 showed, within the limitations of 
the small number of reports used, the value of the Scenario model for clustering reports based on 
similarities of Context plus Outcome. Moreover, the rough codification of the why for this small set 
of reports showed that misrepresentation was a common factor and identified some subjective 
parameters that can be contributing factors to Behavior. This experiment encouraged us to continue 
with our approach to analyzing free text for information on why an incident occurred. 

Then we used our current automated capabilities to cluster the objective parameters as they are 
coded in the current ASRS database. We considered the dominant cluster to be representative of the 
Context of each Scenario, and determined that there are certain common dominant factors associated 
with each anomalous Outcome. We cross-tabulated the data set using ten identified Contextual 
Patterns as the rows and ten chosen anomalous Outcomes as the columns. We then computed the 
ratio between the observed number of observations in each cell and the statistically expected number 
of observations. We concluded that relationships that are both statistically and operationally 
meaningful exist between Contextual Factors/Pattems, on the one hand, and specific types of 
unwanted aviation safety Outcomes, on the other. We recognized that the multiplicity of contextual 
factors that may be present during aviation safety events creates analytical challenges (i.e., the 
dimensionality needs to be reduced through recurrent pattern identification). 

This report has presented a first-generation process for routinely searching large databases of 
aviation accident or incident reports, and consistently and reliably analyzing them for objective 
factors (the what of an incident) as well as the causal factors of human behavior (the why of an 
incident). We have proposed a method for applying the paradigm of Situational Awareness — with its 
five components of Detection, Recognition, Interpretation, Comprehension, and Prediction — to 
automated clustering on the objective and subjective parameters associated with erroneous human 
Behavior from the free-text narrative of an incident report. Noting that the discriminating 
components of SA have a natural sequence, or chronology that can be linked with event chains, we 
have postulated the possibility of identifying effective interventions for the elements of human error 
identified in incident and accident data. 

We have assumed a very simple model for describing the human behavior associated with the 
transition to an anomalous state in our concept of the Scenario. There are likely other factors besides 
loss of SA that could influence transitions in some scenarios. However, the research literature 
documents the very high frequency with which human error can be related to loss of SA. Certainly, 
not all of the contextual factors of the last safe state prevail unchanged throughout the transition, and 
those changes both influence and are influenced by the human actions on the system. Also, it is clear 
that human cognition is a cyclic process and not the simple sequential process of our DRICP 
framework. Nevertheless, we maintain that our simplified model of Scenario and Behavior is both 
necessary and justifiable in this first generation of automated analyses of free text. It is necessary to 
keep the analysis tractable within currently available capabilities, and it is justifiable because there is 
every reason to believe that ASRS reports are usually delivered as sequential narratives. The 
research process will be designed to continuously question our assumptions, and our simplifications 
will be corrected as required through future investigations. 

The plan is to continue to develop and enhance the automated capability to correlate Context and 
Outcome by incorporating additional domain knowledge. For this first-generation process, we 
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believe that it is essential to (1) maximize the information from the objective parameters about what 
happened in order to minimize the domain for analyzing why it happened, and (2) assume a 
simplified model of Behavior to begin to analyze automatically for an understanding of why. In the 
experiment to be conducted during the next year, we will evaluate the ability to automatically extract 
useful information about why a set of similar incidents occurred based on this simplified model. 
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APPENDIX A: CURRENT TOOLS 


Several commercially available tools have been evaluated in queries of the ASRS database: 

BRIO: Enables the performance of a Boolean search (SQL-like) on the relational database 
containing the codified part of the incident. Moreover, it enables a search for words in the free-text 
section. This tool is efficient and widely used for retrospective searches. The user must know how to 
formulate a “good” question in order to use BRIO effectively. 

QUORUM: QUORUM-Perilog is a set of methods and software for data mining of text and other 
sequences of symbols. QUORUM measures the degree of contextual association of large numbers of 
word pairs in narratives and other text to produce models that capture the contextual structure of the 
text. It compares models to measure their degree of similarity. The QUORUM tool is primarily 
dedicated to retrospective searches, rather than analyses. 

Vivisimo Clustering Engine™: Interfaces with any document database to automatically organize 
search or database query results into hierarchical folders of categories that are selected from the 
words and phrases contained in the search results themselves. In a small experiment on a subset of 
ASRS reports, Vivisimo was found to be easy to run and was able to identify operationally pertinent 
concepts and exemplars. However, it was more effective when used on categorical fields than on 
free text, and the preprocessing of vocabulary was an important enabling step. 

Battelle PNWD methodology: A new set of tools has been developed in order to mine the ASRS 
database and build clusters without knowing what we are looking for. This methodology uses 
domain knowledge to standardize the language of the free text for processing (an automated filtering 
process called PLADS), statistical tools to identify clusters and super clusters (Matlab), and a 
software dedicated to navigating the hierarchical structure (called ALAN). The AUTOMATIC 
LANGUAGE ANALYSIS NAVIGATOR (ALAN) is a text comprehension tool that clusters textual 
data. ALAN identifies aviation safety reports that have similar topics, or identifies clusters of reports 
that are similar to a given exemplar (Willse et al. 2002). 

The heart of the ALAN methodology relies on the extraction of a signature for each report. The 
signature and the definition of the similarity between two signatures are based on word counting in 
the free text. Results of clustering are often difficult to understand from an operational perspective, 
they do not provide automated identification of precursors, and they cannot be used to build an 
intervention strategy for a critical situation. Research is currently being conducted to make better use 
of domain knowledge to improve the efficiency and the operational relevance of the clustering tools. 
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APPENDIX B: THE CINQ-DEMI METHODOLOGY 


During the 1980s, a French organization called Cinq-Demi developed a tool for analyzing conditions 
and the operational system faults underlying incidents or accidents (Lecomte et al. 1992, Wanner 
1999). This methodology has been used successfully to analyze accidents in a variety of domains, 
and on selected accident-inducing events reproduced in a flight simulator. In 1992, personnel of 
NASA-FAA’s Aviation Safety Reporting System (ASRS) Office became interested in evaluating the 
potential application of this method to the ASRS database to aid identification of aviation system 
deficiencies. Representatives of the ASRS initiated discussions with ONERA and with Cinq-Demi 
about collaborating on an evaluation of the methodology. 

In 1995, NASA and ONERA agreed to a new task titled “Human Factors in Aeronautical Operations 
and Incidents” under the existing ONERA-NASA Memorandum of Agreement (MO A) for 
collaborative research in aeronautics. The intent of this task was to evaluate the applicability of the 
Cinq-Demi methodology to the ASRS incident database. Consequently, the Cinq-Demi method was 
tested and was found to agree closely with the opinion of ASRS analysts in identifying incident 
causal factors in a sample set of about 300 ASRS reports. 

The underlying concept of the Cinq-Demi methodology is best understood from the perception of 
aviation safety depicted in figure B-l. The status space of figure B-l is an N-dimensional space 
representing all the parameters that define the state of the system. The “Status Point” defines the 
state of the world from the perspective of the aircraft. The operational objective is to maintain the 
Status Point within the “Authorized Flight Envelope” where the probability of an accident is very 
low (say, 10" ). Outside the Authorized Flight Envelope is the “Peripheral Envelope” where the 
probability of an accident is somewhat higher (say, 10 3 ). A trespass into the Peripheral Envelope is 
an incident. In such cases, the task of the operator is to bring the aircraft back from the Peripheral 
Envelope to the Authorized Envelope. When the trespass exceeds the Peripheral Envelope, the 
consequence is a highly probable accident. 


17 , 

There are substantial similarities between the Cinq-Demi methodology and the Human Factors Analysis and 
Classification System (HFACS) (Shappell and Wiegmann 1997, Wiegman and Shappell 2001) that has become well 
known in the US. 
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Figure B-l. A view on accident prevention. 


There are only three types of activities that can influence the movement of the Status Point, and 
these are represented in the first three grids of Cinq-Demi’s structured methodology for analysis. 
One such type of activity is associated with “Maneuverability,” represented in a Cinq-Demi coding 
grid called the “Grid of Aircraft Maneuver Events (GAME)” (fig. B-2). The GAME grid lists the 
maneuvers that are either imposed by the mission, or required for correction of the Status Point to 
accommodate environmental events. 


MANEUVERABILITY 

Maneuvers imposed by the mission 


Speed or Mach Number changes 

Mmm 

Flightpath angle changes (particularly flare) 

Mmp 

Heading changes (turn entry, turn, turn exit) 

Mmc 

Altitudes changes (climb or descent entry, climb or descent, level off) 

Mmh 

Configuration changes (Landing gear, flaps, airbrakes, systems on/off) 

Mms 

Correction maneuvers 

Speed or Mach Number correction 

Mem 

Angle of attack or Longitudinal Attitude correction 

Mci 

Sideslip angle correction 

Med 

Lateral Attitude correction 

Mca 

Heading correction 

Mcc 

Altitude correction 

Mch 


Figure B-2. Grid of aircraft maneuver events (GAME). 


The second type of activity is “Sensitivity to Disturbances,” represented by the “Grid of Aircraft 
Sensitivity to Perturbations (GASP).” (See fig. B-3.) The GASP lists perturbation events that are due 
to internal disturbances (such as a system failure), or external disturbances (such as turbulence or a 
sudden change of runway status) that result in movement of the Status Point. 
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SENSITIVITY TO PERTURBATIONS 

External Disturbances 

Gust 

Sraf 

Wind Gradient 

Sgrv 

Turbulence 

Stur 

Thunderbolt 

Sfdr 

Icing 

Sgiv 

Hail 

Sgrl 

Runway (Rapid change of status: holes, patches of snow...) 

Spst 

Bird 

Soix 

Internal Disturbances 

System Failure 

Span 

Fire 

Sfeu 

Cabin pressure loss 

Sprs 

Disturbance due to passenger 

Spax 


Figure B-3. Grid of aircraft sensitivity to perturbations (GASP). 


The third type of activity is “Pilotability,” represented by the “Grid of Operator Failures (GOOF).” 
(See fig. B-4.) In performing a task, an operator can miss or badly execute an elementary operation. 
GOOF identifies the “Elementary Operations” and the “Error Factors,” i.e., the conditions leading to 
that error. The Elementary Operations are Data Collection, Data Treatment and Decision, Data 
Transmission, and Action. There are five Error Factors: High Workload, Lack of Informational 
Cues, Misrepresentation due to wrong use of information and cues, Misrepresentation due to 
“diabolic” error, and Clumsiness. 
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GOOF 

ELEMENTARY OP] 

ERATIONS 



ERROR 

Data Collection 

Decision after Data 

Data Transmission 

Action 

FACTORS 

Machine=>Man or 

Man=>Man 

*S 

Treatment 

(Diagnostic) 

*d 

Man=>Machine or 
Man=>Man 

*t 

*a 

High Workload 

Saturation : Data not 

Null, partial or wrong 

Saturation : 

Saturation : No 


collected or wrongly 

data treatment (diag- 

Transmission 

action or erroneous 


captured. 

nostic) leading to a bad 

inexistent, 

action on a control. 

c* 

Cs 

decision. Too fast or too 
late decision. 

Cd 

incomplete or 
wrong. 

Ct 

Ca 

Lack of cues 

Lack of vigilance : 

Null, partial or wrong 

Lack of vigilance : 

Lack of vigilance : 

(Under vigilance, 

Data not collected or 

data treatment 

Transmission 

No action or 

very low 

wrongly captured. 

(diagnostic) leading to a 

inexistent, 

erroneous action on 

workload) 


bad decision. Too late 

incomplete or 

a control. 

A* 

As 

decision. 

Ad 

wrong. 

At 

Aa 

Misrepresen- 

Use of a wrong data 

Use of a wrong working 

Use of a wrong 

Use of a wrong 

tation 

collection model 

or risk model : false, 

transmission model 

controls model 

(Model error. 

(localization, 

oversimplified or too 

(localization, 

(localization, 

Wrong use of 

identification or 

complex. 

identification, way 

identification, way 

Data) 

transposition model). 


of action, 

of action, status...) 

M* 

Ms 

Md 

addressee). 

Mt 

Ma 

Misrepresen- 

Data collection 

Changes of situation 

Change of 

Action based on the 

tation 

limited to those 

denied or forgotten. No 

transmission status 

a priori model. 

(A priori model 

which corroborate the 

risk awareness. 

denied or forgotten. 


"diabolic" error) 

p* 

a priori model. 

P 

Pd 

Pt 

Pa 

Clumsiness 

Wrong data 

Misunderstanding 

Wrong transmission 

Erroneous action 


collection by visual 

during data treatment. 

by slip of the 

due to an incorrect 

L* 

or auditory lapse. 

Ls 

Ld 

tongue. 

Lt 

motion of a hand, a 
foot, a finger. . . 

La 


Figure B-4. Grid of operator failures (GOOF). 


To describe an incident or an accident using the Cinq-Demi process, the analyst of a reported 
incident must first list, in chronological order, the sequence of sub-events as reported and then, for 
each sub-event, identify and code the three types of reported activities that can move the Status 
Point, by making the appropriate selections from the GAME, GASP, and GOOF grids. 

Cinq-Demi points out that other factors related to the operator’s physical and psycho-sociological 
behavior can be conducive to human error, but these cannot be resolved through operational or 
technical solutions. Factors such as these must be taken into account only to estimate the probability 
of occurrence of the same situation. They are accounted for in the fourth grid, called the “Grid of 
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Amplifiers of Risk of Errors (GARE).” (See fig. B-5.) The GARE grid is used to identify the human 
environment at the time of the incident. It includes physical factors, physiological factors, 
psychological factors, and sociological factors. 


GARE - Grid of Amplifiers of Risk of Errors 

Physical Factors 

External Factors 

Reduced Comfort 

- Seats, abnormal body position . . . 



Embarrassing 

- Safety suit, gloves, boots, spectacles, 

Pk 

Pke 

working suit 

mask, earphones,. . . 



Work station 

- Vibrations, shakes, low frequency 



motion 

oscillations (air sickness), work under load 
factor. . . 



Environment 

- Temperature (High or Low), cabin 
pressure, humidity, lighting (too low or too 
high), noise, smell... 



Time at incident 

- Mission beginning or end, just back from 



occurrence 

holiday or just before holiday, during 
holiday, schedule changes... 


Internal Factors 

Medicines 



Pki 

Alcohol 

Drugs 


Physiological 


Fatigue 


Factors 


Needs 

- Hunger, thirst, natural needs. . . 



Pathological status 

- Sickness, flu, aches (head, teeth, ears...), 

Pg 



itching, incapacitation (faint, death) 

Psychological 


Fear, Anguish 


Factors 


Personal troubles 
Family troubles, 


Ps 


Memory loss, madness. . . 

Sociological 

Internal Factors 

Crew or team structure 

Factors 

Si 

Crew member’s qualification 
Occasional manpower shortage 

S 


Learner, beginner. . . 




Team internal dispute. . . 


External Factors 

Bad social environment (strike. . .) 


Se 

Visitor, Instructor, Inspector VIP. . . on board 


Figure B-5. Grid of amplifiers of risk of errors (GARE). 


The matrix of Operational System Faults and Elementary Operations constitutes the fifth and final 
coding grid that is called the “Rapid Analysis Fault Table (RAFT).” (See fig. B-6.) Cinq-Demi 
presumes that the error factors identified in the GOOF grid have their roots in systemic faults. The 
RAFT grid is meant to help to analyze these system faults of an incident, and to categorize them 
relative to the following concepts: 

• Organization (crew roles, responsibilities, tasks, and procedures) 

• Design (basic design concept rather than interface) 
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• Design Interface-Mechanical ergonomic (interface design for operation) 

• Design Interface-Mental ergonomic (interface design and interpretation) 

• Education-Training Basic 

• Education-Training Specific (systems, model, and proficiency) 

• Documentation (physical faults) 

• Documentation (wrong content) 

• Requirements (company, regulatory, and equipment manufacturer) 

Within the computerized version of the grids for GOOF and RAFT, there are illustrative examples 
and definitions available at each “cell” within these matrices (accessible by a double-click of the 
mouse on the cell) to aid the analyst in deciding on the appropriate categorization for an event. 


54 



RAFT 

ELEMENTARY OPERATIONS 

OPERATIONAL 
SYSTEM FAULT 

Data Collection 
Machine=>Man or 
Man=>Man 

*s 

Data Treatment 
(Diagnostic) 
Decision 
*d 

Data 

Transmission 

*t 

Action 

*a 

ORGANIZATION 

o** 

ORGANIZATION 
Who has to do ? 
Responsibility 

Or 

ORGANIZATION 
What to do ? 
Carrying out 

Oe 

ORGANIZATION 
With what to do ? 
Means 

Om 

ORGANIZATION 
How to do ? 

Drills Op* 

Ops 

Opd 

Opt 

Opa 

INTERFACES 

INTERFACES 
Mechanical ergonomy 

Hm* 

Hms 

Hmd 

Hmt 

Hma 

INTERFACES 
"Mental" ergonomic 

He* 

Hcs 

Hcd 

Hct 

Hca 

EDUCATION 

p** 

EDUCATION- 

TRAINING 

Basic education Fb 

Fb 

EDUCATION- 

TRAINING 

Specific educ. Fs* 

Fss 

Fsd 

Fst 

Fsa 

DOCUMENTA- 
TION D** 

DOCUMENTATION 
Physical faults 

Dm 

Dm 

DOCUMENTATION 
Wrong content 

Dc* 

Dcs 

Dcd 

Dct 

Dca 


REQUIREMENTS 

R* 

Rs 

Rd 

Rt 

Ra 


Figure B-6. Rapid analysis fault table (RAFT). 
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APPENDIX C: TAXONOMIC STRUCTURE FOR CODIFICATION 


In the example presented in Section 3.2 (see fig. 2), states and events have been described in an 
informal way by sentences or words extracted from the narrative. Choosing the particular set of 
parameters that describe each state of the world, and the particular set of parameters that describe 
each transition, will give us a more formal description of each part of the Scenario of such an 
incident. 

A wide range of taxonomic structures is used in the various accident/incident databases. These 
structures often contain parts related to the description of the flight circumstances, and others that 
relate to the human factors of an event. As an example, O’Leary et al. (2002) gives a flavor of the 
type of parameters used in the British Airways Safety Information System (BASIS), while 
Murayama and Yamazaki (2002) show some of the Performance Shaping Factors used in a marine 
incident reporting system. The basis of our study relies on a consolidation of the three taxonomic 
structures that underlie three codifications 19 that were designed specifically for use with ASRS 
reports: 

• The ASRS codification is a structured set of ‘descriptors’ that is currently used to describe 
the incident and store it in the database. The codification is designed for use by operational 
personnel. It is limited, primarily, by the size of the current paper-reporting form that the 
ASRS Office uses for the sake of maintaining confidentiality. After 28 years of operation, 
well over 100,000 ASRS incident reports have been codified with this taxonomic structure 
and are available in the ASRS database. 

• The X-Form is another template that has been designed for the codification of ASRS reports. 
It contains additional descriptors that are intended to address human-factors issues that had 
not been considered in the design of the original ASRS codification, but have since become 
of high interest. The X-Form has never been implemented for routine operation in the ASRS. 

• The Cinq-Demi methodology (described in Appendix B) was developed during the 1980s as 
a tool for analyzing aeronautical-incident reports from a human-factors point of view. This 
methodology focuses on identifying conditions that have a high probability of leading to 
human errors. In 1997, a codification form was designed, built upon the ASRS codification, 
but with additional fields to make the codification more compatible for efficient search and 
analysis using the Cinq-Demi methodology. Small sets of ASRS reports have been codified 
using this tool and are available. 

As highlighted in Wiegmann and von Thaden (2003), most incident reports are highly informative 
about what happened but give much less definitive information about why an incident happened. In 
a first-level filtering of a clustering process, we need to be able to cluster incident reports reliably on 
the basis of similarities of what happened. Our assumption is that this step can be achieved by the 
use of the taxonomic structure. 


18 

By taxonomic structure we mean a set of structured terms that describes some domain or topic as in Swartout et al. 
(1997). The idea is that a taxonomic structure provides a skeletal structure for a knowledge base. 

Codification refers specifically to the attributes and their values that constitute the fixed fields of the reporting form. 
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Therefore, our first objective was to identify all the possible terms of the taxonomy and their 
structure for the incident reports in our world of aviation. Each term will map to a parameter in the 
description of our world in the incident model (described in Section 3). Moreover, for every 
parameter used, we can state whether the concept captured is objective or not. We will call a concept 
objective if it can be defined on the basis of observable data. All the concepts that are not objective 
are called subjective. 

Our hypothesis is that a “full and complete” set of objective parameters adequately describes the 
what and could be incorporated into the fixed fields of a well-designed computerized reporting form 
for electronic submission. Then the first step of the clustering process (on similarities of the what) 
could be totally automated. The understanding of the why will rely on subjective parameters and on 
exploitation of the free text. 

In Appendix D, we will discuss the merger of all the parameters identified in the three codifications, 
and we will assume that the result constitutes a “full and complete” set of parameters for the 
description of any and all aviation incidents in our world. However, as these three codification 
schemes entail structured sets of descriptors, we will, in this appendix, first compare these 
codifications at the highest levels of their structures. 

C-l. High-Level Structure of the Three Codifications 

In this section, we are going to study separately the taxonomic structures of the three codification 
forms for the ASRS reports. We will classify the type of information contained in the main sections 
of the three forms into the following five categories as the highest level of their structures: 

1. Time and Setting: We group in this category all the information related to the frame of the story 
(when, where...) and to the fixed entities (facilities, equipment...). 

2. Cast of entities: This category contains information on the persons and all the entities that 
evolve and take action in order to create the story. 

3. Anomaly: This pertains to all the information that explains why the “anomalous state” is 
anomalous. 

4. Transitions: This is all the information that characterizes a transition in the State/Transition 
model. 

5. Other: This includes any information that cannot be classified in any of the other four 
categories. 

The main purpose of these five categories is to better understand the main similarities and 
differences among the three codifications in their structures and their relations to the State/Transition 
model. Some sections of a codification form may address several categories, and then we will go one 
step down into the knowledge structure to understand their differences. 

For each codification form, we are also going to highlight the codified links between the main 
sections of their structures. Our focus in this study on the high-level structure and linkages will, of 
course, not reveal the relations between pieces of information at lower levels in the taxonomic 
structure. 
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C-l.l ASRS Codification 

The ASRS fields are organized into the following nine main sections: 

1. Time: The time section gives the date, the day, and the local time of day for incident occurrence. 
The local time is given only as a six-hour time interval, and we can assume for almost all 
incidents that the entire story occurs during this interval of time. This section of the ASRS 
codification is part of our “Time and Setting” category. 

2. Place: The place section contains 4 subsections (Locale Reference, State Reference, Relative 
Position, and Altitude). We can assume that the entire story is linked to the same State Reference 
and Local Reference. However, the Relative Position and the Altitude subsections describe a 
very precise point of the space, and that point is identified with the anomalous state reached 
during the story. This section is in the “Time and Setting” category. 

3. Environment: This section describes the weather, light, visibility, ceiling, and runway visual 
range (RVR) and falls in the “Time and Setting” category. In the spirit of the ASRS codification, 
the environment section relates to the weather conditions that are generally presumed not to 
change during the course of events of the reported incident. Nevertheless, some incident reports 
describe a rapid deterioration of the weather conditions, and in such cases these parameters could 
vary. 

4. Aircraft: Aircraft are, of course, central elements in an aviation incident. Their descriptive 
parameters evolve with time, and the incident report often describes their different states. We 
point out that their description in the ASRS codification contains fields that are not intrinsic 
characteristics of the aircraft but are linked with other “elements” of the story (e.g., Controlling 
Facilities, Coordinating Facilities, etc.). These elements will be studied in section C-2. For the 
most part, an aircraft is considered as an entity in the ASRS codification, and therefore this 
section falls within the “Cast of entities” category. 

5. Component: A component is one part of an aircraft. The link between the component and the 
aircraft is well codified in the ASRS form. The ASRS form puts the component in a separate 
section for historical reasons. For the purpose of this discussion, we consider the component as a 
part of the related aircraft entity. This section falls into the “Cast of entities” category. 

6. Person: People are the other essential entities of the story. This section also falls in the “Cast of 
entities” category. Some of the subsections of the ASRS form are used to link the described 
person to other entities (e.g., aircraft). 

7. Events: The events section describes several things, including the anomalous state(s) 
encountered in the story and the following actions that resulted in recovery to a safe state. 
Consequently, some of the subsections of the ASRS “Events” are in the “Anomaly” category, 
others are in the “Transitions” category, while some belong to the “Other” category. 

8. Maintenance Factors: The maintenance factors section is dedicated to incidents occurring 
during maintenance operations. It is not within the scope of this study, and this section is omitted 
from further consideration. 

9. Assessments: The assessments section is mainly an expert’s judgment about the main factors 
that caused this world to reach the anomalous state. It emphasizes some parts of the other fields 
used to describe the story (e.g., an aircraft, a person, an environment factor, etc.). It is a 
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judgment about the importance of some particular part of the description and so falls in the fifth 
category for purposes of this discussion. 

Figure C-l shows the contributions of these sections of the ASRS codification form to the categories 
of the information matrix and to the descriptions of the components of the Incident Model. 



State n 


Time 


Figure C-l. The categories of information, the sections of the ASRS form, and the incident model. 


As already mentioned, some links between parts of knowledge are explicitly codified in the ASRS 
form. Others are only implicit. Figure C-2 shows the links that are both explicit and completely 
defined in the ASRS form. 



Figure C-2. Explicit links in the ASRS form. 
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Other more or less implicit links also exist in the ASRS database. We can point out three types of 
such links: 

• Links between parameters describing the anomaly and one or several entities in order to 
describe an anomalous state (for example, an anomalous state described by “Ground 
Incursion” should be linked to an aircraft). These links are often implicit. 

• Links between an event and one or several entities. For example, the Resolutory Action 
“Diverted to Alternate” is associated with the Flight Crew in the ASRS codification (because 
“Diverted to Alternate” is an entry in the “Flight Crew” subsection). Nevertheless the link 
between the event and the entity is only partly explicit as the story might contain several 
flight crews. 

• Links between entities and other entities that are “not well defined.” For example, the sub- 
section, “Controlling Facilities,” links the aircraft with something (tower, TRACON...) that 
is not clearly codified in the ASRS form (there are no descriptors for the tower, 

TRACON...). 

C-1.2 The X-Form 

The X-Form is another template that has been developed for the codification of ASRS reports. It was 
designed to overcome some of the shortcomings of the ASRS form that had been identified after 
several years of use. The X-Form has never been implemented at ASRS. The definition of the 
required or possible links between the main sections, and how these links are codified, is not always 
clear. 

The X-Form is organized into the following 18 sections: 

1. Record Control: This groups information that enables good management of the database 
(accession number, type of codification, etc.). This section is not directly related to the 
description of the story and falls in the “Other” category. 

2. Time: “Time and Settings” category. 

3. Place: This is similar to the Place section in the ASRS Form. It is a part of the “Time and 
Setting” category. It contains both a general description of the location and a precise description 
of the place of the anomalous state. 

4. Environment: This section describes the weather, visibility, terrain, etc., and is part of the 
“Time and Settings” category. 

5. Traffic: This describes the overall air/ground traffic at the time of the story. It is part of the 
“Time and Settings” category. 

6. Airspace: This describes the Airspace involved in the story. Therefore, at least one aircraft 
should be linked to each airspace that is involved, and each aircraft involved should be linked to 
at least one airspace. The notion of Airspace is similar to the notion of Place (it is a division of 
the space) and falls in the “Time and Settings” category. 

7. Facility-Arpt: An airport is a fixed entity (as is the “Ground” in the Cinq-Demi codification) 
and so is part of the “Time and Setting” category. 
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8. F acility-NA VAID : As for the airport, it is a fixed entity and a part of the “Time and Setting” 
category. 

9. Facility-ATC: As for the airport and the NAVAID, it is a fixed entity and a part of the “Time 
and Setting” category. 

10. Aircraft: Falls in the “Cast of Entities” category. It contains both static and dynamic parameters. 

11. Person: The person section falls in the “Cast of Entities” category, but the HUMPERF (Human 
Performance) subsection of Person describes actions made by the person. So the HUMPERF 
subsection is part of the “Transitions” category. 

12. Info-Probs: This section describes events linked to a communication problem. This specific type 
of event has been added in the X-Form as it seems to entail crucial steps leading to an anomalous 
state. These events are always linked to at least one person. It falls in the “Transitions” category. 

13. Conflict: This section describes the anomalous state and so is part of the “Anomaly” category. 

14. Adverse Interaction: This section contains three parts: Interpersonal, Proximity, and 
Coordination. The second one, Proximity, characterizes the proximity of the airspace to an 
airport and can be considered as part of the “Time and Settings” category. The two other parts 
describe adverse interactions between persons or a coordination failure. They are more related to 
the description of the sequence of events and are classified in the “Other” category. 

15. Event Flow: This section describes both the anomalous state (“Anomaly” category) and the 
following events (“Transition” category). 

16. Situation: This section is codified only for reports related to a recurrent event or to a situation 
(the “same” situation has already been encountered several times in the last months). Its aim is to 
identify a “latent fault” in the system (policy, procedure, etc.) and is a little like some elements 
of the RAFT in the Cinq-Demi codification. It falls in the “Other” category. 

17. ATC: This section contains two parts. The first one, ATC-HANDLING, describes actions taken 
by the ATC and is part of the “Transition” category. The other one, AIR TRAFFIC INCIDENT, 
mixes anomalous state descriptions (e.g., NMAC) and judgments about the role of the persons 
(PFT-DEV, INTERCOORD...). We are going to put this second subsection in the “Other” 
category (partly because, in the ASRS form, the Air Traffic Incident subsection was in the 
ASSESSMENTS section.). 

18. General Assessment: This is a judgment about the importance of some factors already codified 
and so falls in the “Other” category. 

Figure C-3 shows the relationships of the sections of the X-Form to the categories of information 

and to the descriptions of the Incident Model. 
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Figure C-3. The categories of information, the X-Form sections, and the incident model. 


Even if the links between the sections of the X-Form are not always clearly codified, we should have 
at least the ones shown in figure C-4. 



Figure C-4. The links between the sections of the X-Form. 
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There are other links included in the codification as, for instance, between the Info-Probs section and 
other sections; this is because a communication problem can entail an equipment problem. We will 
discuss later how to represent the elements of the “Transition” category in order to retain all the 
necessary links. 

C-1.3 Cinq-Demi Codification 

A description of the Cinq-Demi methodology is presented in Appendix B, and the structure of the 
Cinq-Demi codification designed for use on ASRS reports has already been extensively reviewed in 
previous studies (Maille 2001a and 2001b). The Cinq-Demi codification contains mainly three sorts 
of information: the description of the frame of the story (“Time and Settings” category), the 
description of the entities (“Cast of Entities” category) and the descriptions of events related to the 
persons (“Transitions” category). The Cinq-Demi codification primarily relies on two types of 
entities: aircraft and person. Their possible links are well identified and formalized. The “Ground” 
environment is highlighted in the Cinq-Demi formal codification, and some equipment problems can 
be linked to it. The “Ground” refers to the airport facilities and equipment and so we consider it to 
be part of the “Time and Setting” category. 

Cinq-Demi’ s field called “Theme” is sometimes an “Anomaly” description and sometimes a 
“Transition” description. The confusion over the meaning of this field was, in fact, the origin of the 
discussions that led to the scenario concept described in this report, and so it will not be considered 
further. 

Figure C-5 shows the relationships of the sections of the Cinq-Demi codification to the categories of 
information and to the description of the Incident Model. 
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Figure C-5. The categories of information, the sections of the Cinq-Demi codification, 

and the incident model. 


An important difference between the ASRS and the Cinq-Demi codifications is that the persons 
encoded are not exactly the same. The Cinq-Demi codification identifies only the categories of 
persons involved in an error rather than the individuals. 
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C-2. Comparison of the Three Structures 

Each one of the three structures has some special characteristics, but their high-level organizations 
are quite similar. We must keep in mind that these forms are used to codify the report primarily for 
efficient retrospective search of the database and not as the sole basis of an in-depth analysis of the 
incident. That is why they include neither a precise description of the sequence of transitions, nor an 
accurate time reference of events or states. We are now going to describe the differences among the 
three forms for each category of information, and we will define the taxonomic structure that seems 
to be the most relevant for purposes of our study. Our objective is to define a taxonomic structure 
that supports both efficient retrospective search and the in-depth analyses that are the subject of this 
report. The sections used in the ASRS codification are all included in the X-Form’s sections. The 
Cinq-Demi codification introduces two new sections (GOOF and RAFT) for the description of the 
transitions. 

• “Cast of entities”: The three codifications do not entail exactly the same entities, but all three 
refer to the notions of Person and Aircraft. 

- Person: The Cinq-Demi codification highlights only groups of persons that have made 
some error, and their codification is designed for understanding conditions that lead to 
human error. As the ASRS reports are primarily used for an operational analysis of 
incidents, it is certainly better to encode all the persons involved in the incident. 
Nevertheless, we will have to be sure that our codification allows us to identify which 
persons have made an error. (The links between the fields used to describe the error or 
the conditions leading to the error, and the person or team responsible for the error, must 
be clearly codified.) 

- Aircraft: The only difference among the three forms is that the ASRS codification has a 
separate section for the description of an aircraft component. We point out that the 
information related to the component is often more related to a description of an event 
(malfunctioning, failed, improperly operated. . .) than it is to a description of the aircraft. 
As we want to highlight the transitions (the why and the how), we propose to group such 
information with the other transitions (as in the Cinq-Demi grids where the Technical 
Failure (Span) is classified as one of the possible perturbations). 

• “Time and Setting”: The most complete description is the one used in the X-Form. We are 
going to use the sections of the X-Form as a starting point for this category. 

• “Transitions”: In our analysis of the three forms, we put the information related to the 
“Transitions” (at least the one involving a person) in a separate category, but in the templates 
that information can be: 

- incorporated in another section (for instance HUMPERF is a subsection of the PERSON 
section), or 

- in a specific section (e.g., INFO-PROB section). 

Incorporating the transition information into another section enables us to relate it to the 
other information of that section. This is used to highlight two sorts of links: 
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- The link between the transition and the person responsible for that transition (for instance 
HUMPERF subsection). Thus the actor performing the action is unambiguously 
identified. 

- The link between the transition and a specific state. (For instance, the “Resolutory Event” 
subsection of the ASRS form is in the same section as the “Anomaly” and the 
“Independent Detector” subsections, highlighting their link.) 

Incorporating the transition information within a person description raises two problems: 

- It does not allow a good codification of an event linked to several persons (such as 
communication problems). 

- It does not allow an easy representation of the sequence between the events (even if we 
do not know yet whether we really want to codify such a sequence in a codification 
process) or an easy retrieval of how things happened. 

As we assume that a “good” schemata that captures the essence of the story of the incident is that of 
the Scenario (i.e., Context + Behavior -> Outcome), we propose to clearly separate in the taxonomic 
structure all the information related to the transitions and, therefore, to Behavior. The “Resolutory 
Action/Event” must be identified separately (as it is done in the X-Form and in the ASRS form) 
because it is not part of the “Behavior” of our Scenario. For our current study, the Outcome of the 
Scenario is an anomalous state and the Behavior of recovery to a safe state is not a part of this study. 
The Scenario that includes consideration of recovery from an anomalous to a safe state will be the 
subject of a subsequent study. 

• “Anomaly”: This is described in a very similar way in the ASRS and in the X-Form. The 
two descriptions will be merged for our structure. 

• “Other”: Having a separate section for record control seems to be a good idea. The other 
sections (e.g., Assessments, Situation) have to be consolidated depending on the aim of our 
codification. 

The overall taxonomic structure of the knowledge and its links with the three parts of the Scenario 
are shown on figure C-6. 

C-3. A “Full and Complete” Set of Parameters 

In section C-2, we compared, at a high level, the taxonomic structures that underlie the three 
codification forms specifically designed for ASRS reports. We showed that the structure of the 
knowledge used in the ASRS codification is embodied in the one used by the X-Form. Therefore, we 
will use the X-Form structure on which to map all the parameters used in all three forms. Parameters 
that are addressed only in the Cinq-Demi form will be inserted at the most relevant place in that 
structure. 

The set of all the parameters of this taxonomic structure is given in Appendix D. 
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Figure C-6. Main relations between the Scenario and the categories of information. 
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APPENDIX D: MAPPING PARAMETERS TO A “FULL AND 

COMPLETE” SET 


This appendix describes the process of formulating a “full and complete” set of parameters based on 
a consolidation of those that have been identified in (1) the current ASRS codification form, (2) the 
Cinq-Demi GRIDS, and (3) the “X-Form.” All of these parameters have been classified according to 
the five categories of information (Time and Setting, Entities, Anomaly, Transitions, and Other) and 
their sub-categories as shown in figure 6. A separate table of parameters is built for each sub- 
category. 

Moreover, this set of factors has been separated into the following two categories: 

1. Factors that are clearly and unarguably objective, categorical, and measurable (for simplicity, 
these will be called “objective” factors in this presentation). 

2. All the others (that, for the moment at least, we will simply label as “subjective”). 

We need not seek “perfection” in this process of classifying parameters as objective or subjective. 
Certainly, there are factors that everyone would put in category 1 that could have an aspect of 
subjectivity (such as visibility, for example). Certainly, there will be differences of opinion on where 
to assign some factor, but we do not consider this degree of uncertainty to be very important to the 
result because the number of questionable factors constitutes only a small portion of the total 
number of parameters. 

In each of the following tables, the titles of the tables should not to be confused with the fields. The 
fields (or the attributes) associated with each title are in yellow cells. The parametric values of these 
attributes are shown in white cells if they are objective parameters or in green cells if they are other 
than objective (called subjective) factors. 
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Time and Setting 


Time 


Place 

Date of Occurrence 


State Ref. 

Day of Occurrence 


Local Ref. 

SUN 


Facility 

MON 


ID 

TUE 


Type 

WED 


ARPT 

THU 


VOR 

FRI 


VORTAC 

SAT 


NDB 

Time of Occurrence 


TACAN 

0001 to 0600 


Intersection 

0601 to 1200 


Special Use Airspace 

1201 to 1800 


Relative Position 

1801 to 2400 


Distance 


Radial 


Angle 


Altitude 


MSL 


AGL 
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Environment 

Flight Conditions 

Lighting 

VMC 

Dawn 

IMC 

Daylight 

Mixed 

Dusk 

Special VFR 

Night 

Marginal 

Visibility in Statute Miles 

Ceiling 

Low Boundary (statute miles) 

CLR 

Upper Boundary (statute miles) 

Single Value (feet) 

Single Value (statute miles) 

Lower Boundary (feet) 

Runway Visual Range 

Upper Boundary (feet) 

Lower Boundary (feet) 

WX-AVD 

Upper Boundary (feet) 

Weather Elements 

Single Value (feet) 

Wind Fx 

Terrain Fx 

CLR-TURB 

MTNS 

WAKE 

HILL 

TURB 

RISING 

ALOFT 

DITCH 

SHEAR 

TREE 

HEAD 

WIRE 

TAIL 

TOWER 

CROSS 

OTH-OBST 

TSTORM 

WATER 

DN-DRAFT 

GRADE 

UP-DRAFT 

CRANE 

Obscur'n Fx 

BRIDGE 

SMOG 

BUILDING 

PRECIP 

VEHICLE 

DUST 

OBS-LTG 

CLOUDS 

Other Fx 

SUNPOS 

BIRDS 

UNDCAST 

ANIMAL 

OVERCAST 

PED 

HAZE 

LTNG 

OBSTRUC 

LITE 

FOG 

SIGN 

WDW 

MARK 

Precip'n Fx 

ENGICE 

RAIN 

FRAMICE 

TSTORM 

FOB 

DRIZZLE 

SKYDIVER 

SLEET 

BARO-GRADIENT 

SNOW 

WXBAL 

HAIL 

RAPID-DETER 


JETBLAST 
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Facility Airport 


Facility ATC 

Involvement 


Involvment 

SURF 


CTRL 

O&D 


NBRQ 

COM 


COM 

PROB 


PROB 

Facility State 


Facility Type 

Facility ID 


TWR 

Facility Descriptors 


TRACON 

PVT 


ARTCC 

CTL 


MILFAC 

UCTL 


FSS 

CLOSED 


CPNY-RDO 

SATELLITE 


CTAF 

CONSTRUCTN-ACTIVITY 


UNICOM 

RWYS-IN-US E-PAR 


Facility State 

RWYS-IN-US E-INTER 


Facility ID 

RWYS-IN-US E-CVG 


Facility Descriptors 

RWYS-IN-US E-DVG 


DARC-ACTIVATED 

RWY-CHG-IN-PROGRESS 


BUEC-ACTIVATED 

CRASH-ACTIVATED 


TRNG-IN-PROG 

HELIPORT 


CRASH-ACTIVATED 

SEAPLANE 


CLOSED 

Problem Components & Services 


NON-RDR 

RWY 


NON-FED 

TXWY 


Problem Components & Services 

RAMP 


RADAR 

SURFACE 


COM-ECP 

WX-EQP 


COM-ENV 

COM-EQP 


OTH-EOP 

COM-ENV 


COMPUTER 

INTXN-NAME 


STRUCTURE 

DMEN 


SRVCS 

SRVCS 


STAFF 

PROC-POL 


PROC-POL 

STAFF 


SCOPE 

MGMT 


MGMT 


Traffic 

OPPDIR 

UNKVFR 

SAMEDIR 

POPUP 

SIDEBY 

UNAUTH 

CONVERG 

PLTDEV 

INTERSEC 

FLTASSIST 

PARALLEL 

EMER 

SAMEALT 

SPC-EVENT 

OVERTAKE 

BOUNDARY 

PERFDIFF 

TFC-SEQ 

CONGEST 

CROSSING 

FLYWAY 

FORMATION 

OVERFLT 

FORM-BREAKUP 

NORAC 

FORM-JOINUP 

NORDO 

CLOSURE-RATE 
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Airsf 

oace 

Involvement 

Routes (cont'd) 

OCCU 

Arrival 

SHUD 

Profile Descent 

ENT 

Holding Pattern 

EXIT 

STAR 

PROB 

On Vectors 

Airspace ID 

VFR 

Type 

Approach 

Class A 

Circling 

Class B 

Contact 

Class C 

Instrument Precision 

Class D 

Instrument Non Precision 

Class E 

SVFR 

Class G 

Traffic Pattern 

Special Use 

Visual 

Temporary Use 

Charted Visual 

SUA (Special Use Airspace) 

Straight-In 

PROHIB 

Military 

RESTR 

T ransit 

REFUEL 

OverWTR 

WARN 

IAPS 

SR 

ILS 

ALERT 

VOR 

MOA 

NDB 

VR 

MLS 

IR 

PARALLEL 

OSUA 

RNAV 

DZ 

SDF 

ROUTES 

TACAN 

Departure 

Design Problem 

SID 

MAP-PT 

Noise Abatement 

INTXN-NAME 

Other Published IFR Departure 

APCHES 

On Vectors 

DEPS 

Enroute 

CHARTING 

Airway 

XING-ALT 

Direct 

PROX 

On Vectors 

HMDG 

Atlantic 


Pacific 


Other Oceanic 
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Facility NAVAID 

Involvement 

NAV-ERR 
COM 
PROB 
Facility Type 
ILS 
VOR 
VORTAC 
NDB 
TACAN 
BCSTN 
LDIN 

ROT-BEAC 
MLS 
LORAN 
SATELLITE 
OMEGA 
OTH-VIS 
Facility State 
Facility ID 

Problem Components & Services 
ILS 

COM-EQP 

COM-ENV 

LITE 

SRVCS 

STRUCTURE 


Adverse Interactions 

Proximity Btwn Terminals & Airspc 
Civil-Mil Arpt 
Civil-Civil Arpt 
Route-Terminal 
Route-SUA 
Bird Flyway-Terminal 
Canadian Airspace 
Mexican Airspace 
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Entities 


Aircraft 

Make Model 

Advanced Ckpt 

Aircraft Type 

DISPLAY 

SMA 

NAVCTL 

SMT 

NON 

LTT 

Operator Organization 

MDT 

Common Carrier 

MLG 

Air Carrier 

LGT 

Air Taxi 

HVT 

Charter 

WDB 

General Aviation 

FGT 

Corporate 

BMB 

Instructional 

MLT 

Personal 

MTR 

Other 

SPC 

Government 

ULT 

Military 

SPN 

RNT 

BAL 

Mission 

HNG 

Passengers 

OTH 

Freight 

Crew Size 

T raining 

1 

Pleasure 

2 

Agriculture 

3 

Ambulance 

40M 

Ferry 

Airframe 

T est Flight 

wings 

T actical 

WL 

Refueling 

WM 

T raffic Watch 

WH 

Other 

WB 

Banner T ow 

WO 

Business 

WR 

Photo-Shoot 

gear 

Repositioning 

LN 

Skydiving 

LR 

EMS 

LF 

CBO 

surf mod 

CKD-RID 

SL 

FLT-CHK 

SS 

UTL 

SA 

PRB 

SI 

Flight Plan 

engines 

VFR 

ER 

IFR 

ET 

DVFR 

EJ 

SVFR 

EN 

COM 

Number of Engines 

NON 


Aircraft 

(cont'd) 

Fit Phase 

Fit Phase (cont'd) 

GND 

ARR 

PREFLT 

DSCNT 

PUSH-BK 

APCH 

POWER-BK 

LNDG 

TAXI 

HLD-SHT 

GND-HOLD 

AIR-HOLD 

HLD-SHT 

MNTN 

INTXN-XING 

PATTERN 

Parked 

GAR 

Maintenance 

SHT-FLD 

Holding 

OFF-ARPT 

Position and Hold 

MAP-PT 

Takeoff Roll 

PARK 

DEP 

LOW 

TKOF-POS 

TAG 

TKOF 

INTXN-LNDG 

ABORT 

DN-WIND 

INTXN-TKOF 

SIDESTEP 

ICLB 

Intermediate Altitude 

CLB 

Vacating Altitude 

DN-WIND 

Roll 

SHT-FLD 

Missed Approach 

OFF-ARPT 

Operating Under FAR Part 

MITO 

Part 91 

Intermediate Altitude 

Part 119 

Vacating Altitude 

Part 121 

CRS 

Part 125 

WX-AVD 

Part 129 

VECTOR 

Part 135 

DIV 

Other Part 

DIRECT 

Maneuver 

Level 

Imposed by Mission 

Holding 

Speed 

Enroute Altitude Change 

Angle 

MNV 

Heading 

TURN 

Altitude 

180 

Configuration 

360 

Correction 

AUTO-ROT 

Speed 

LLL 

Angle Attack/Longi. 

LLH 

Sideslip 

TOW 

Lateral Attitude 

OTH 

Heading 

EMER 

Altitude 


Aircraft 

cont'd) 

Routes 

Approach 

Departure 

CAT-1 

SID 

CAT -II 

Noise Abatement 

CAT -III 

Other Published IFR Departure 

BC 

On Vectors 

LDA 

Enroute 

MLS 

Airway 

PAR 

Direct 

RNAV 

On Vectors 

SDF 

Atlantic 

TACAN 

Pacific 

OHD 

Other Oceanic 

OPPDIR 

Arrival 

IN IT 

Profile Descent 

FINL 

Holding Pattern 

SHT-FINL 

STAR 

NON-RDR 

On Vectors 

PRACTICE 

VFR 

DN-WIND 

Approach 

UP-WIND 

Circling 

SIDESTEP 

Contact 

Cabin Activity 

Instrument Precision 

Beverage Service 

Instrument Non Precision 

Boarding 

SVFR 

Cart Service 

T raffic Pattern 

Deplaning 

Visual 

Meal Service 

Charted Visual 

Movie 

Straight-In 

Safety Related Duties 

Military 

Seated 

T ransit 

T ray Service 

OverWTR 

Other (activity) 

Navigation in Use 

Cabin Lighting 

ILS 

Bright 

Localizer Only (RWY ID) 

Medium 

Localizer & Glideslope (RWY ID) 

Dark 

VOR 

Available Seats 

NDB 

Pass, on Board 

FMS or FMC 

Maintenance Status 

GPS 

Required Documentation on Board 

INS 

No 

Loran 

Yes 

Pilotage 

Released for Service 

DME 

No 

Descriptors 

Yes 

FGN-FLAG 

FORMATION 

ORDNANCE-ON-BOARD 

Maintenance Deferred 
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Aircraft 

(cont'd) 

Problem Systems 

ATA Code 

Placard/Marking 

Manufacturer 

Servicing 

Problem 

Air Conditioning 

Design Deficiency 

Auto Fit 

Failed 

Communications 

Improperly Operated 

Electric Power 

Malfunctioning 

Eqp/Furnishings 

Not Installed 

Fire Protection 

Aircraft Problems 

Fit Ctls 

CKPT-NOISE 

Fuel 

MTNC-DISCREPANCY 

Hyd raulic Power 

DECOMPRESSION 

Ice/Rain Protect 

FIRE 

Indicating/Recording Sys 

ACFT-PERF 

Landing Gear 

PREFLT-DEICING 

Lighting 

ENG-OUT-PERF 

Navigation (+ FMS) 

CONFIGN 

Oxygen 

FLEET-INCONSIST 

Pneumatic 

DESIGN 

Vacuum 

CABIN-SPACE 

Water/Waste 

VIS-SIGNATURE 

Electrical Panels & Parts 

INSPECTION 

APU 

CREW-COMPLEMENT 

Doors 


Fuselage 


Nacelles/Pylons 


Stabilizers 


Windows 


Wings 


Propeller 


Rotor 


Pwe Plant 


Eng Fuel Ctl 


Ignition 


Air 


Eng Ctls 


Eng Ing/Warning 


Exhaust 


Oil 


Starting 


T urbines 


Water Induction 



Person 

Person's Involvement 

Function at Tme of Occurrence (cont'd) 

Pilot Flying 

Controller 

Pilot Not Flying 

Local, Combined Local 

Monitoring 

Local, Combined Local 

Controlling 

Ground, Combined Ground 

Cabin Service 

Ground, Combined Ground 

Evaluating 

Flight Data 

Instructing 

Clearance Delivery 

Receiving Instruction 

Departure 

Maintenance 

Approach 

Observing 

Radar 

Other Direct Involvement 

Combined Radar 

Affiliation 

Non Radar 

Government 

Handoff Position 

FAA 

Traffic Management 

Foreign 

Flow 

Military 

Maintenance 

Company 

Inspector 

Air Carrier 

T echnician 

Air Taxi 

Lead T echnician 

Charter 

Flight Attendant 

Corporate 

On Duty 

Other 

Off Duty 

Contracted Service 

Extra 

Instructional 

MILFAC 

Personal 

PAR 

CGA 

RANGE 

NGA 

RSU 

UGA 

OTHER 

Function at Time of Occurrence 

FSS Specialist 

Oversight 

UNICOM Operator 

PIC 

FBO Personnel 

Supervisor 

Vehicle Driver 

Coordinator 

Dispatcher 

Flight Attendant in Charge 

Gate 

Airport Manager 

Ramp Guidance 

Observation 

CGP 

Air Carrier Inspector 

CENR 

Company Check Pilot 

Qualifications 

Observer 

Pilot 

Passenger 

Student 

Instruction 

Private 

Instructor 

Instruments 

T rainee 

Multi-Engine 

Flight Crew 

Commercial 

Single Pilot 

ATP 

Captain 

CFI 

First Officer 

Flight Engineer 

Second Officer 

Military 

Relief Pilot 


Navigator 


Load Master 
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Person 

(cont'd) 

Qualifications (cont'd) 

Factors Adversely Affecting Perf. 

Controller 

Physical Factors 

Military 

External Factors 

Radar 

Reduced Comfort 

Non Radar 

Workspace 

Developmental 

Seats 

SPI 

Abnormal Body Position 

T echnician 

Under Hood 

Repairman 

Embarrassing Working Suit 

Powerplant 

Safety Suit 

Airframe 

Gloves 

FCC 

Boots 

Inspection Authority 

Earphones 

Flight Attendant 

Work Station Motion 

Currently Qualified 

Vibration, Shakes 

T rainee 

Low Freq. Oscillations 

Aircraft Qualified on (number) 

Load Factor 

Other 

Rate of Climb 

FSS Specialist 

Rate of Descent 

Dispatcher 

Environment 

Experience 

Temperature 

Controller 

Cabin Pressure 

Radar 

Humidity 

Non Radar 

Lighting 

Supervisor 

Background Contrast 

Military 

Audio Interferences 

Limited Radar 

Noise 

Time Certified in Pos 1 

Visual Interferences 

Time Certified in Pos 2 

Smell 

General/Total 

Smoke 

Dispatch 

Time at Incident Occurrence 

FSS Specialist 

Mission Beginning 

Flight Attendant 

Mission End 

Total 

Back from Vacations 

Airline Total 

Before Vacations 

Type 

During Vacations 

Maintenance 

Schedule Changes 

Repairman 

Duty Cycle Length 

Technician 

Shift-Chg 

Lead T echnician 

Internal 

Avionics 

Medicines 

Supervisor 

Alcohol 

Flight Time 

Drugs 

Total 


Last 90 Days 


In Acft Type 
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Person 

(cont'd) 

Factors Adversely Affecting Perf. (cont'd) 

Factors Adversely Affecting Perf. (cont'd) 

Physiological Factors 

Sociological Factors (cont'd) 

Fatigue 

External Factors 

Needs 

Bad Social Environment (Strike) 

Hunger, Thirst 

Visitor 

Natural Needs (toilets) 

Instructor 

Pathological Status 

Inspector 

Sickness, Flu 

VIP 

Aches 

CPNY-BUS 

Itching 

Psychological Factors 

Obvious Incapacitation 

Fear, Anguish 

SUB Incapacitation 

Personal T roubles 

Vertigo 

Personal Preoccupations 

Hypoxia 

Family T roubles 

Illusion 

Memory Loss 

Yehudi 

Madness 

Black-Hole 

TASKLOAD 

White-Out 

SGL-PLT 

Sloping-Ter 

COMBO-POS 

Disorientation 

COMBO-SEC 

Sociological Factors 

CREW-COMPLEMENT 

Internal Factors 

PREOCC 

Crew or T earn Structure 

TFC 

Qualification 

WX-AVD 

Unqualified 

EQP-PROB 

Not-Current 

TRNG-IN-PROG 

Occasional Manpower Shortage 

FLTASSIST 

Team Internal Dispute 

CHKLST 

Proficiency 

TUNING 

In-Doubt 

OTH-TASK 

Learner, Beginner 

SPC-EVENT 

T raining Deficiency 

FUEL 

Recency-of-Experience 

ATTITUDE 

Language Barrier 

UNPROFESS 

Familiarity 

ANTAGON 

ARPT 

COMPLACENT 

ATC-PROC 

GETHOME 

EQP 

AGGRESS 

AREA 

RSCE-DEFIC 

TERRAIN 

CTLR 

ACFT-PERF 

SUPVR 

WX 

FLO 

AIRSPACE 

CHART 

REGS 

PUB 

NAVAID 

FSS 

ARPT-PROC 

NAVAID 

CHART 

ACFT-EQP 

NEW 

ATC-EQP 

ATC-SRVC 


OTH 


PAX-DISCOMFORT 

PAX-ILLNESS 
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Anomaly 


Anomaly 

T echnical 

Loss of control 

Acft Eqp Prob 

Ground Excursion 

Critical 

Rwy Overrun 

Less Severe 

Rwy Excursion 

ATC Com (lost or intermittent) 

Txwy Excursion 

False/Erratic Course Indie. 

Ramp (Gnd Excursion) 

Fuel 

Other 

Exhaustion 

Gnd Loop 

Inadequate 

Stall 

Contamination 

Spin 

Type 

Hyd roplane 

Encounters 

Maintenance Problem 

In-Fit Encounter 

Improper Maintenance 

VFR in IMC 

Non Compliance with MEL 

VFR over the TOP 

Improper Documentation 

Weather 

Tkof 

Turbulence 

Overweight Tkof 

Birds 

Landing 

FOB 

Overshot 

Obstruc 

Undershot 

Skydivers 

Gear Up Landing 

Wake Turbulence 

Tailstrike 

Gnd Encounter 

Hard Landing 

FOB 

Wrong Rwy 

Ped 

Wrong Arpt 

Animal 

Landing Without Clr 

Birds 

Txwy Lndg 

Eqp 

Overweight Lndg 

Jet Blast 

Apch 

Vehicle 

Wrong Rwy 

Deviations 

Wrong Arpt 

Speed Dev 

Unstabilized Apch 

Alt Dev 

Ground Incursion 

Overshoot on Clb 

Taxi 

Overshoot on Dscnt 

Runway 

Undershoot on Clb 

Conflict 

Undershoot on Dscnt 

Airborne 

Excursion Clb 

NMAC 

Excursion Dscnt 

Air Less Severe 

Xing Restrict Not Met 

Ground 

Acft at Imprud Alt 

Gnd Severe 

Descent Below/MSA 

Gnd Less Severe 

Other Spatial Deviation 

Airspace Violation 

T rack or Hdg Dev 

Unauth Incursion 

Acft on Imprud Track 

Unauth Excursion 

Control. Fit Toward Terrain 

Uncoord Penetration 

Unctrl Arpt Tfc Pattern Dev 

Uncoord Exit 

Altitude Heading Rule Deviation 


Glideslope 
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Anomaly (cont'd) 

Non Adhere Legal Rqmt 

Cabin Event 

Clrnc 

Galley Fire 

Pub Proc 

Passenger 

MEL 

Misconduct 

Wx Mins 

Illness 

FAR 

Contraband 

Alt-Hdg 

Electronic Device 

Alt-Setting 

Other 

Ster Ckpt 

FLC Status 

Speed 

Hazardous Cargo Problem 

inspec 

Smoke or Fire 

AD 

Fumes 

Company Policies 

CG Irregularity 

Required Legal Separation 

Uncoord 

Non Comp/Srvc Advsy 

Sector Penetration 


Rwy Movement 


Conflict 

T raffic Mix 

ACFT 
VEH 
VSL 
OTH 
Fit Regime 
GND 
TMNL 
ENRTE 
Event Severity 
MIN 
MOD 
NEAR 
Miss Distance 

Horizontal Miss 
Vertical Miss 

Diag/Unspect'd Miss 


Transitions 


Problematic Human Performances 

Data Collection/Transmission 

Decision after Treatment 

COM 

PLAN 

INTRA-CKPT 

PREFLT 

INTER-CKPT 

INFLT 

ATC-FLC 

TFC 

CPNY-FLC 

APCH 

INTER-FAC 

BACKUP 

INTRA-FAC 

TFC-SEQ 

CKPT-CAB 

DECIDE-Y 

GND-CKPT 

DEP 

TECHNIQ 

TKOF 

OBSERV 

ABORT 

TFC 

RTN 

SPACING 

AVD-WX 

ALT 

GAR 

HZ-POS 

DIVERT 

ATT 

COMPLY 

SPEED 

DEVIATE 

WX 

TIMING 

EQP-STAT 

EMER 

MIS-ID 

LNDG 

TECHNIQ 

MAP 

ARPT 

CROSS 

LANDMARK 

HLD-SHT 

RWY 

MTN 

TXWY 

CLIMB 

INTXN 

DSCNT 

WALK-AROUND 

TURN 

TGT 

AIRHLD 

CHK-PT 

TKOF-POS 

LOST-SIGHT 

EXER-COM-AUTH 

ATTITUDE 

DH 

HEADIN 


HEADOUT 


SCAN 


UTILIZE 


FLC 


ATC 


EQP 


FSS 


CHART 


PUB 


PF 


PNF 


SO 


DMAN 


SUPVR 


CTLR 


FLTSTRIP 


NAVAID 


STAGE3 
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Problematic Human Performances (cont'd) 


Decision after T reatment (cont'd) 

DECIDE-N 

DEP 

TKOF 

ABORT 

RTN 

AVD-WX 

GAR 

DIVERT 

COMPLY 

DEVIATE 

TIMING 

EMER 

LNDG 

MAP 

CROSS 

HLD-SHT 

MTN 

CLIMB 

DSCNT 

TURN 

AIRHLD 

TKOF-POS 

EXER-COM-AUTH 

DH 

MANAGE 

PRIORITIZE 

ASSIGN 

DELEGATE 

DISCARD 

RQST-INPUT 

RSPND-INPUT 

EVALUATE 

INITIATE 

MONITOR 

FAIL-INTERV 

TERMINATE 

EARLY 

LATE 

DEV-SOP 

INFLEX 

INDECISION 

UNLOAD 

RECOGNIZE 


Action 

EQP-USE 

PROGRAM 

SWITCH 

TBLSHOOT 

SETUP 

HANDLE 

MODE-SEL 

TECHNIQ 

NAV 

INSTRUM 

DEADRECK 

PILOTAGE 

TECHNIQ 

CHK-PT 

MANIP 

TKOF 

CRS 

APCH 

LNDG 

TAXI 

PATTERN 

IMC 

CROSS-WIND 

TECHNIQ 

OTHER 

DEICING 


Information Problems 

Message Origin 

Type 

PSN 

DATA 

FLC 

ALT 

ATC 

HZPOS 

GNDCREW 

SPEED 

CAB 

ATT 

FSS 

AIRSPACE 

EQP 

REGS 

ACFT 

TERRAIN 

ATC 

WX 

SCOPE 

ARPT 

TAPE 

ROUTE 

CHART 

PROC 

SECTIONAL 

BRAKING 

WAC 

FLT-PLAN 

PLATES 

STORED 

IFR-ENRTE 

HYDRAULICS 

PUB 

FUEL 

ATC-HD-BOOK 

ACS-STATUS 

AIM 

ELECTRICAL 

FARS 

PRESSURIZATN 

FOM 

WT-BAL 

MEL 

ADVSY 

NOTAM 

CALLOUT 

BULLETIN 

POINTOUT 

LOA 

TFC 

DIRECTIVE 

WX 

OTH 

AIRMET 

NOTES 

SIGMET 

FLT-PROG-STRIP 

PIREP 

DISPATCH 

CHKLIST 

LOG 

CHALLENGE 

Destination 

RESPONSE 

PSN 

SCUAWK 

FLC 

1200 

ATC 

7500 

GNDCREW 

7600 

CAB 

7700 

FSS 

INSTRUC 

OTH 

CLRNC 

Media 

AMENDED 

DIR 

CANCELLED 

VIS 

EXPEDITE 

AUD 

TAPED 

GESTURE 

ATIS 

GND-ECP 

TWEBS 

TPHONE 

FEED-BK 

IPHONE 

CONFIRM 

INTER 

READBACK 

INTRA 

ACKNOW 

COMPUTER 

CUERY 

RDO 

ROST 

CTAF 

PERMSN 

UNICOM 

PROC 

WRITTEN 

FREO-CHG 

LOG 

LOST-COM 


Information Problems (cont'd) 

Type (cont'd) 

Problems (cont'd) 

COORD 

EQP 


HDOFF 

GND 

BRIEFG 

AIR 

MANUAL 

OTS 

INTENT 

WEAK 

OTH 

INTERMITTENT 

IN-BLIND 

ENVIRON 

ABBRV 

FREQ-BLOCK 

RELAY 

SIMUL-XMISSN 

Reference Phase 

RANGE 

GND 

STEEPED-ON 

DEP 

NOISY 

CRS 

AUD-INTERF 

ARR 

VIS-INTERF 

MNV 

FREQ-CONGEST 

OTH 

FREQ-LAP 

Problems 

RECEPTN 

CONTENT 

WRNG-FREQ 

FALSE 


NOT-OBS 

INCOMPL 


NOT-MON 

AMBIG 


NOT-HEARD 

MISLEAD 


MISSED 

IMPRUD 


INADEQUATE-DISSEM 

CONFUS 

INTERP 

TIMING 


MIS-INTERP 

NEVER 


MINDSET 

EARLY 


CALLSIGN 

LATE 


NAME 

OTH-PRIORITY 


NUMBER 

XPRESSN 


LANGUAGE-BARRIER 

SIM-SND 

RESPONSE 

SIM-LOOK 

DENIED 

PHRASEOLOGY 

REFUSED 

TRANPOS 

NON-COMPLY 

MISSTATE 

UTILIZN 

CALLSIGN 

FORGOT 

NAME 

DEGRADED 

NUMBER 

OTH 


VOX-QUAL 

GARBLED 

ENNUNC 

INTERMITTENT 

CLUTTER 

LOST-COM 

WRNG-FREQ 


SPCH-RATE 


LANGUAGE-BARRIER 
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ATC-HANDLING 

Clrnc 

Coord 

Imprud 

Untimely 

Misdirected 

Imprud 

Misstated 

Misstated 

LTSS Permitted 

Misdirected 

Uncoord 

Non 

Sector Penetration 

Briefing or Relief 

Rwy Movement 

Fit Prog Strip 

Advsy 

Not Posted 

Safety Not Sent 

Late Posting 

Tfc Not Sent 

Not Marked 

Wx Not Sent 

Improper Mrkng 
Not Scanned 

Nonstandard Phraseology 
Fit Plan Handling 


Misrepresentation 

Ext. Environment Mod 
Flight Mechanics Mod 
Load. /Spec. Eqpt Mod 
MELMod 

3D T rajectory Mod 
Risk Mod 
Structure Mod 
Press/AirCond Sys Mod 
Flight Auto Sys Mod 
FMS Mod 

Communicat. Sys Mod 
Electrical Sys Mod 
Auxiliary Eqpt Sys Mod 
Flight Controls Sys Mod 
Fuel Sys Mod 
Hydraulic Sys Mod 
Rain/Ice Sys Mod 
Landing Gear Sys Mod 
Navigation Sys Mod 
Oxygen Sys Mod 
Aux. Power Sys Mod 
Powerplant Sys Mod 


Operational System Fault 

Organization 

Responsibility 
Carrying Out 
Means 
Drills 

Design 

Basic Design 
Mechanical Ergonomy 
Mental Ergonomy 
Education 

Basic Education 
Specific Education 
Documentation 

Physical Faults 
W rang Content 

Requirements 
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Resolutory Event 


Independent Detector 
Cockpit 

Flight Crew 
Aircraft Equipment 
Altitude Alert 
TCAS 
GPWS 
Lite 
Vox 
Sound 


Resolutory Action (cont'd) 


ATC 

Controller 
ATC Equipment 
MSAW 
Conflict Alert 
OEDP 

Resolutory Action 
Controller 

Intervened 

Issued New Clearances 
Declared Emergency 
Ordered Expedited Man. 

Gave DF Steer 
Activated Crash Alert 
Separated T raffic 
Provided Flight Assist 
Issued Advisory 
Issued Alert 

Aircraft 

Automation Overrode Fit Crew 
Equip. Prob. Dissipated 
None Taken 
Unable 

Anomaly Accepted 
Detected After-the-Fact 
Insufficient Time 


Flight Crew 

Avoid-Evas Action 
Exec. GAR or Missed Apch 
Regained Acft Control 
Overcame Equip Problem 
Aborted Tkof 
Became Reoriented 
Ret. Original Clrnc/Course 
Exec. Lost Com Procedure 
Declared Emergency 
Perf. Expedited Maneuver 
PNF Interv/Seized Controls 
PNF Interv/Other 
Exerc. of Command Autho. 
Prepared for Ditching 
Executed a 180 
Executed a 360 
Dumped Fuel 
Man. Out Penetrated Airsp 
Man. Out Adverse Env 
Abandoned Apch 
Returned to Land 
Diverted to Alternate 
Forced Landing 
Landed Off Arpt 
Ordered Evacuation 
Precautionary Landing 
Evacuated (Aircraft) 

Landed as Precaution 
Diverted to Another Airport 
Returned to Assigned Airspace 
Returned to Assigned Altitude 
Landed in Emergency Condition 
Overrode Automation 
Fire Extinguished 
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Other 


Record Control 

Accession Number 
Coding Form 
Coding Status 
Multiple Report Flag 
SGL 
MUL 

Analyst Graphic 
Y 
N 

Source 

NF 

ANONOM 
Reporter's ACN 
Receipt Date 
Reporter Graphics 
Response to Reporter 
Analyst Callback 
Completed 
Attempted 
None 


Air Traffic Incident 

Air T raffic Incident 
NMAC 

Operational Error 

Operational Deviation 

Other ATC Handling 

Pilot Deviation 

Military Facility Deviation 

Intra Facility Coordination Failure 

Inter Facility Coordination Failure 

Declared Emergency 

FLT -ASSIST 

SPILL-IN 

SPILL-OUT 
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General Assesments 

Type of Event 

Problem Areas 

Unique Event 

Performances 

Recurrent Event 

Flight Crew Human Performance 

Unwanted Situation 

ATC Human Performance 

Special Handling 

Cabin Crew Human Performance 

ROU 

Maintenance Human Performance 

ABR 

Passenger Human Performance 

OHN 

Procedures-Policies 

FYI 

ATC 

TEL 

ARPT 

RCC 

ACR 

MMW 

MIL 

Primary Problem 

Company 

ATC Human Performance 

FAA 

Cabin Crew Human Performance 

Documentation 

Flight Crew Human Performance 

Chart 

Passenger Human Parformance 

Publication 

Aircraft 

Regulation 

ATC Facility 

Design 

Airport 

Airspace Structure 

Navigational Facility 

Aircraft 

Airspace Structure 

Airport 

Company 

Equipment 

FAA 

ATC 

Chart or Publication 

NAVAID 

Environmental Factor 

Aircraft 

Weather 

Environmental Factors 

Ambiguous 

Weather 

Special Educational Value 

IFE 

Y 

Preoccupation 

N 

Diversion 

T raffic 

Ranked Severity 

Conflict 

Number (0 to 1 0) 

Preoccupation 
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Situation 

GEN-SIT 

FAA-POL 

Acft Type or Class 

See and Avoid 

Aircraft Design 

Fac Staffing 

Equipment 

Keep 'em H igh 

Airframe 

Enforcement 

Engine 

Flow Control 

Chart 

Noise Abatement 

Publication 

Non Radar Proc 

An Intxn Name/Other Name 

Criteria 

Design 

Wx Observ 

Airspace 

Separation 

Route 

Certification 

Physical Facility 

FLC 

ATC 

Ctlr 

Arpt 

Equipment 

NAVAID 

Oth Psn 

TFC Performance Mix 

Info Dissemination 

Procedure or Policy 

Charted Procedure 

ATC Fac 

Arr 

Arpt 

Dep 

Company 

Other 

FAA 

Twr-Enroute Ctlr 

NAVAID 

Equipment 

FAR 

T ransponder 

CPNY-POL 

Oth 

Alt Callout 

T raining 

Alt Rptg 

FLC 

Crew Scheduling 

Oth 

Mntc Scheduling 

Air Carrier Inspection 

Push-Bk 

Sterile Ckpt 

Pwr-Bk 

Deicing 

Loading Procedure 
Info Dissemination 
Emergency 
Schedule Adherence 
Fuel Conservation 

Emergency 


Adverse Interactions 

Interpersonal 

Coord, within and between Facilities 

Intra-Ckpt 

IntraFac 

Inter-Ckpt 

Twr 

Intra-Fac 

TRACON 

Inter-Fac 

ARTCC 

Supvr-Ctrl 

FSS 

Ctrl-Oth 

MilFac 

FLC-ATC 

InterFac 

FLC-CAB 

Twr-Twr 

FLC-GND 

Twr-TRACON 

FLC-DISP 

Twr-ARTCC 

FLC-FSS 

Twr-FSS 

FLC-CENR 

Twr-MilFac 

FLC-CHKPLT 

TRACON-TRACON 

FLC-Oth 

TRACON-ARTCC 

Oth-Oth 

TRACON-FSS 

Labor Relations 

TRACON-MilFac 

ARTCC-ARTCC 

ART CC-FSS 

ARTCC-MilFac 

FSS-FSS 

FSS-MilFac 

MilFac-MilFac 


Consequence 

FAA/ATC 

Investigated 

Assigned or Threatened Penalties 
Reviewed Incident with Fit Crew 

Other 

Physical Injury 
Emotional T rauma 
Acft Damaged 
Tail Skid 
Wing Tip 
Undercarriage 
Propeller 
Tires 
Fire 

Company Review 
Flight Canceled 
Stranded 

Maintenance Action 
Violation Not Pursued 


91 



APPENDIX E: SITUATION AWARENESS 


E-l. Introduction 

In this report, we are concerned with failures of Situation Awareness, and it is important that this be 
differentiated from a similar term that is often used in related literature, Situation Assessment. 
Situation Assessment is used in contexts that are slow, deliberate, strategic, and search-oriented. 
Situation Awareness is used in the context of events, processes, and interactions that are fast, event- 
driven, and tactical or reactive. 

Situation Assessment would be an appropriate term to use, for example, if a management team were 
considering a large, long-term investment in a foreign country, perhaps the construction of new 
manufacturing facilities. They would be well advised to gather data and to carry out systematic 
statistical projections related to such topics as employment patterns, vocational performance, 
education, technology development, drug abuse, disease, and security. They might even have an 
opportunity to employ sophisticated data-fusion methods and other problem-solving techniques to 
place their decisions and actions on a rational footing. 

Situation Awareness (SA) is concerned with a completely different set of issues: the operational 
state of an expert human performer in a dynamic and potentially dangerous environment. In this 
report, we are considering pilots and air-traffic controllers operating in the global civil aviation 
environment. Other studies of SA have focused on challenging military operations, such as 
command and control in joint-operations combat. Still others have studied automobile drivers, 
anesthesiologists, space mission ground-controllers, and firefighters. 

In Section 9 of this report, we proposed that Behavior in our definition of Scenario is always 
associated with loss of SA. To make progress on the analysis and measurement of Behavior, it is 
necessary to break SA down into more concrete and constructive components. Fortunately, we can 
draw on an extensive SA research literature. A number of previous studies have highlighted levels or 
stages of SA that are closely related to our list of discriminating components: Detection, 

Recognition, Interpretation, Comprehension, and Prediction (DRICP). Furthermore, we can draw on 
the extensive research literature in related domains of human factors, expert performance, and 
behavioral decision theory. 

These components of SA can be further defined as follows: 

Detection is the act of discovering, discerning, or capturing attention as this is related to the 
existence, presence, or fact of an event. To be detected, event E must entail a change above 
threshold or a change from adaptation level, though E does not have to be assigned to a more 
abstract class or type. Balakrishnan (1998) provides a discussion of detection and the 
important related concept of vigilance. The vigilance-detection paradigm can be seen as the 
most elementary setting for Situation Awareness in which adequate performance is defined 
simply as noticing and responding to changes from baseline stimulation. (We note that 
simple does not mean easy — pure vigilance tasks are notoriously difficult and error-prone.) 
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Recognition is the act of relating a detected event E to a class or type of event that has been 
perceived before. Event E can be assigned to an event type when it is perceived as a 
recurrence of something experienced previously. Richman et al. (1996) discuss the central 
importance of recognition in expert performance. Expertise, in general, and expert-level 
Situation Awareness, in particular, depend on the acquisition of reliable and nearly automatic 
domain- specific skills of recognition. 

Interpretation is the act of relating a specific event type to a network of actual and possible 
events of various other types. Event E cannot only be assigned a class, but it can also be 
related to other classes of event types within a conceptual structure. Ericsson and Kintsch 
(1991) have described these kinds of conceptual structures as long-term working memory 
(LTWM) and have presented experimental evidence for the central role of LTWM in 
experts’ interpretation of domain- specific events. 

Comprehension is the act of perceiving the significance of an event. Event E not only can be 
assigned a place within a logical or categorical paradigm, but can also be understood in terms 
of its role in a familiar temporal pattern of events. The pattern may enable an expert to infer 
past events that must have caused E, or future events that must follow from E, or concurrent 
events that must accompany E. 

Prediction is the act of forecasting what will happen in the near future. Event E is understood 
as part of a predictable sequence, so that specific future events are expected based on the 
occurrence of E. In many domains, including aviation, experts typically stay “ahead of the 
curve” by actively predicting and preparing for plausible continuations. 

The aim of this Appendix is to provide pointers into the research literature on SA, and to explain 
briefly how our technical approach, described in Section 9, relates to previous work. We will also be 
as explicit as possible about the boundary conditions and limitations of our approach, 
acknowledging that our simplifications will have to be corrected through future research. 

E-2. Situation Awareness, Prediction, and Active Cognition 

Humans are limited in the amount and kinds of information they can process, and in the speed with 
which they can process it. Highly trained professionals — pilots, physicians, firefighters — can get 
into situations in which the apparent information-processing requirements exceed human abilities. 
Yet experts usually perform reliably in these kinds of environments. How is this possible? 

The lowest level of SA involves detecting and recognizing low-level attributes and dynamics of 
objects and events. The second level involves interpreting and comprehending the situation based on 
knowledge of significant, but more abstract, relations among the recognized elements. This level of 
interpretation and comprehension relates concrete objects and events to operational goals in ways 
that go beyond the data that are concretely available. The third level of prediction requires the ability 
to project the near-term course of events into the future. This highest level achieves closed-loop 
behavior via continuous perception of situation elements in relation to goals, threats, resources, 
actions, and consequences. 

The key factor in many types of expert performance seems to be what Jones and Endsley (1996) 
have called "Level 3 SA," that is, prediction or mental projection, " a very demanding task, which 
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people generally perform poorly” (p. 508). Expert knowledge can be defined, in large part, as a set 
of pre-compiled memory structures and specialized cognitive retrieval processes that, together, 
implement predictions of likely event- sequences, including adaptive responses. There is no mystery 
to the pre-compilation process: it is simply the outcome of many years of formal training and 
professional experience (Ericsson 1996, pp.10-11; Richman et al. 1996, pp. 172ff). 

Prediction is the acid test of a scientific theory and is one of the major goals of applied science. 
Elaborate causal models and statistical methods are used to try to predict earthquakes, storms, 
climate change, and the time-course of epidemics. Prediction is also at the core of active learning 
strategies that are advocated by instructors to improve students’ skills in reading, listening, 
mathematics, and test-taking. The ability to predict has obvious practical value in avoiding or 
mitigating the effects of unfavorable events, but it is also instrumental in effective cognitive 
performance. An orientation toward active prediction is the hallmark of cognitive engagement in an 
on-going task. Thus, prediction is a key component of all kinds of individual and collective 
expertise, and is perhaps the most important theoretical link between research on SA and research on 
learning and expertise. 

E-3. Situation Awareness Research 

Jeannot (2000) (see also van Gool et al. 2002) summarizes the Human Factors research on SA, 
emphasizing cognitive aspects such as mental models, long term memory, working memory, 
workload, and human- automation interaction, with special emphasis on the nature of SA in air traffic 
control (cf. Gronlund et al. 1998). He states: 

“In many, if not all, of the controller cognitive models, maintaining Situation Awareness is 
the core sub-process, the basic background activity to air traffic control. The importance of 
“background activity” is recognized as critical by (controllers) themselves. They refer to this 
phenomenon as “having the picture. ” For controllers, “having the picture ” is the first pre- 
requisite to handling their traffic.... “ Losing the picture” is reported as one of the biggest 
risks for controllers, as it is the source of several risks: The controller 

• is no longer able to predict the evolution of the situation, 

• fails to detect early enough a problem or a conflict, 

• does not choose the optimum resolution, 

• and, in extreme cases, allows the creation of incidents or accidents. ” 

Domain-specific representations help to maintain SA via “cognitive economy” (Endsley 2000a). 
Only when information on position and altitude are insufficient for conflict detection will controllers 
look for other sources of information, and then they operate in predictive mode, anticipating the 
situation and working ahead. The more experienced the controller, the more selective his or her 
mental model becomes, even to the point of being “inaccurate” or “distorted” — but in ways that 
promote effective performance (Gronlund et al. 1998). Less experienced controllers have more 
concrete details available about traffic. They tend to “focus on every aircraft,” whereas experts 
classify aircraft into two groups: “those requiring further analysis and those which can be separated 
safely immediately.” 
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Patel et al. (1996, pp. 130ff.) provide an instructive discussion of active information-seeking in the 
context of medical expertise. They describe four stages of information processing called 
observations, findings, facets, and diagnosis. Observations are raw data at our detection/recognition 
(D/R) level. Findings are interpretations (I) of data, and facets are clusters of related findings. 
Diagnosis corresponds to our comprehension (C) level and constitutes the basis for predicting (P) the 
future (prognosis). Facets play an active role in organizing multiple competing interpretations, 
directing the search for additional data (findings) to resolve pending issues, and providing the 
building blocks for a satisfactory diagnosis. In-depth exploration of expert performance shows that 
even the cyclical model is an over- simplification (cf. Frederiksen and White 1990). The more highly 
skilled the expert performer, the more flexibly he or she moves among different knowledge 
representations and different levels of processing. 

Different levels of proactive engagement in the dynamic control task characterize different levels of 
expertise. Cognitive engagement, prediction, and effective control are interrelated. This is one of the 
reasons why automation can sometimes undermine SA, producing 

• loss of vigilance 

• increase in complacency 

• change from active to passive processing 

• loss of or a change in the type of feedback. 

Automation can also become the object of SA itself, in that more experienced operators develop skill 
in predicting the future behavior of automated systems (Jodlowski et al. 2002). Operators’ SA 
regarding automated systems is in turn influenced both by training (Endsley and Robertson 2000) 
and by display design (Kelley 2002). Understanding automation can be defined, in part, as the 
avoidance of “automation surprises.” Avoiding surprise is the same as being able to predict what 
will happen next if a certain input is provided to the automation (Woods et al. 1994). This human- 
automation interaction is further complicated by the fact that the automated system has its own SA 
of the state of its portion of the world. 

In addition to automation effects, the management of SA in aviation and other environments can be 
complicated by factors such as distributed roles and responsibilities, which create the need for 
shared SA (among multiple human and non-human agents), and by mobility of the agents who must 
maintain this shared SA. Artman and Garbis (1998) and Stroeve et al. (2003) show how an initially 
safe situation can evolve into an unsafe one via divergent SA among the operators of the system. 
Johnston et al. (1997) emphasize the need for process analysis in addition to outcome measurement 
(i.e., the why in addition to the what in the terminology of this report). Individual and team processes 
interact in determining overall SA. 

Taking an even broader perspective, Woods et al. (1994) (cf. Cook and Woods 1994) and Moray 
(1994) place SA in the context of human-systems analysis and research on human error as a systems 
problem. Cook and Woods point out the multifaceted nature of SA — control of attention, mental 
simulation, directed attention, contingency planning, mental bookkeeping. In general, any 
multitasking environment requires shifts of attention among different threads, and coordinating these 
shifts requires a coherent system-model or situation-model. 
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E-4. The Cyclic Nature of SA 


In our simplified model of Behavior, we use the DRICP framework of SA as though Detection, 
Recognition, Interpretation, Comprehension, and Prediction occur in sequential order, each 
successive stage using the output of the preceding stage. However, as Carroll et al. (2001) document, 
citing Neisser (1976), human cognition is a cyclic process in which prediction facilitates 
comprehension and interpretation, and in which comprehensible and interpretable events are more 
easily detected and recognized than are unpredictable and incomprehensible events. In fact, Jones 
and Endsley (1996) point out that many Level 2 SA errors (for example, misinterpretation of 
landmarks) can be attributed to incorrect expectations (erroneous predictions), which then cause a 
persistent misrecognition and misinterpretation of perceptual data. 

Figure E-l from Neisser (1976), which we borrow from Carroll et al. (2001), illustrates the more 
complex model of active information seeking. Experts’ skilled performance (Richman et al. 1996), 
as well as their characteristic susceptibility to certain kinds of errors (Cook and Woods 1994; Jones 
and Endsley 1996), can be best understood as a knowledge-driven, prediction-oriented cognitive 
process, not as a data-driven, passive, perceptual one. 



Figure E-l. The perception-action cycle (Neisser 1976). 
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